# DatasetGenerationError while loading dataset

**URL:** <https://discuss.huggingface.co/t/datasetgenerationerror-while-loading-dataset/58841>\
**Category:** Beginners\
**Created:** [October 17, 2023, 8:16am UTC](https://discuss.huggingface.co/t/datasetgenerationerror-while-loading-dataset/58841 "2023-10-17T08:16:04Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![xllcrn](https://avatars.discourse-cdn.com/v4/letter/x/4491bb/32.png) [@xllcrn](https://discuss.huggingface.co/u/xllcrn)\
**Post date:** [October 17, 2023, 8:16am UTC](https://discuss.huggingface.co/t/datasetgenerationerror-while-loading-dataset/58841/1 "2023-10-17T08:16:04Z")

</div>

Hello!  
I’m trying to download dataset verbrannter/invoice\_dataset\_large\_cleaned\_2 and get DatasetGenerationError: An error occurred while generating the dataset during the Generating train split.  
Can you help me?  
Regards

## Downloading data files: 100% 2/2 [00:00\<00:00, 64.17it/s] Extracting data files: 100% 2/2 [00:00\<00:00, 21.97it/s] Generating train split: 0% 0/25434 [00:00\<?, ? examples/s]

FileNotFoundError Traceback (most recent call last)  
File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\datasets\builder.py:1916, in ArrowBasedBuilder.\_prepare\_split\_single(self, gen\_kwargs, fpath, file\_format, max\_shard\_size, job\_id)  
1915 try:  
 → 1916 writer = writer\_class(  
1917 features=self.info.features,  
1918 path=fpath.replace(“SSSSS”, f"{shard\_id:05d}“).replace(“JJJJJ”, f”{job\_id:05d}"),  
1919 writer\_batch\_size=self.\_writer\_batch\_size,  
1920 storage\_options=self.\_fs.storage\_options,  
1921 embed\_local\_files=embed\_local\_files,  
1922 )  
1923 try:

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\datasets\arrow\_writer.py:334, in ArrowWriter. **init** (self, schema, features, path, stream, fingerprint, writer\_batch\_size, hash\_salt, check\_duplicates, disable\_nullable, update\_features, with\_metadata, unit, embed\_local\_files, storage\_options)  
329 self.\_path = (  
330 fs\_token\_paths[2][0]  
331 if not is\_remote\_filesystem(self.\_fs)  
332 else self.\_fs.unstrip\_protocol(fs\_token\_paths[2][0])  
333 )  
 → 334 self.stream = self.\_fs.open(fs\_token\_paths[2][0], “wb”)  
335 self.\_closable\_stream = True

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\fsspec\spec.py:1241, in AbstractFileSystem.open(self, path, mode, block\_size, cache\_options, compression, \*\*kwargs)  
1240 ac = kwargs.pop(“autocommit”, not self.\_intrans)  
 → 1241 f = self.\_open(  
1242 path,  
1243 mode=mode,  
1244 block\_size=block\_size,  
1245 autocommit=ac,  
1246 cache\_options=cache\_options,  
1247 \*\*kwargs,  
1248 )  
1249 if compression is not None:

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\fsspec\implementations\local.py:184, in LocalFileSystem.\_open(self, path, mode, block\_size, \*\*kwargs)  
183 self.makedirs(self.\_parent(path), exist\_ok=True)  
 → 184 return LocalFileOpener(path, mode, fs=self, \*\*kwargs)

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\fsspec\implementations\local.py:315, in LocalFileOpener. **init** (self, path, mode, autocommit, fs, compression, \*\*kwargs)  
314 self.blocksize = io.DEFAULT\_BUFFER\_SIZE  
 → 315 self.\_open()

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\fsspec\implementations\local.py:320, in LocalFileOpener.\_open(self)  
319 if self.autocommit or “w” not in self.mode:  
 → 320 self.f = open(self.path, mode=self.mode)  
321 if self.compression:

FileNotFoundError: [Errno 2] No such file or directory: ‘C:/Users/xllcr/.cache/huggingface/datasets/verbrannter\_\_\_invoice\_dataset\_large\_cleaned\_2/default-b6d7d315edb1e6d7/0.0.0/0111277fb19b16f696664cde7f0cb90f833dec72db2cc73cfdf87e697f78fe02.incomplete/invoice\_dataset\_large\_cleaned\_2-train-00000-00000-of-NNNNN.arrow’

The above exception was the direct cause of the following exception:

DatasetGenerationError Traceback (most recent call last)  
Cell In [2], line 2  
1 from datasets import load\_dataset  
----\> 2 dataset = load\_dataset(“verbrannter/invoice\_dataset\_large\_cleaned\_2”)

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\datasets\load.py:2153, in load\_dataset(path, name, data\_dir, data\_files, split, cache\_dir, features, download\_config, download\_mode, verification\_mode, ignore\_verifications, keep\_in\_memory, save\_infos, revision, token, use\_auth\_token, task, streaming, num\_proc, storage\_options, \*\*config\_kwargs)  
2150 try\_from\_hf\_gcs = path not in \_PACKAGED\_DATASETS\_MODULES  
2152 # Download and prepare data  
 → 2153 builder\_instance.download\_and\_prepare(  
2154 download\_config=download\_config,  
2155 download\_mode=download\_mode,  
2156 verification\_mode=verification\_mode,  
2157 try\_from\_hf\_gcs=try\_from\_hf\_gcs,  
2158 num\_proc=num\_proc,  
2159 storage\_options=storage\_options,  
2160 )  
2162 # Build dataset for splits  
2163 keep\_in\_memory = (  
2164 keep\_in\_memory if keep\_in\_memory is not None else is\_small\_dataset(builder\_instance.info.dataset\_size)  
2165 )

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\datasets\builder.py:954, in DatasetBuilder.download\_and\_prepare(self, output\_dir, download\_config, download\_mode, verification\_mode, ignore\_verifications, try\_from\_hf\_gcs, dl\_manager, base\_path, use\_auth\_token, file\_format, max\_shard\_size, num\_proc, storage\_options, \*\*download\_and\_prepare\_kwargs)  
952 if num\_proc is not None:  
953 prepare\_split\_kwargs[“num\_proc”] = num\_proc  
 → 954 self.\_download\_and\_prepare(  
955 dl\_manager=dl\_manager,  
956 verification\_mode=verification\_mode,  
957 \*\*prepare\_split\_kwargs,  
958 \*\*download\_and\_prepare\_kwargs,  
959 )  
960 # Sync info  
961 self.info.dataset\_size = sum(split.num\_bytes for split in self.info.splits.values())

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\datasets\builder.py:1049, in DatasetBuilder.\_download\_and\_prepare(self, dl\_manager, verification\_mode, \*\*prepare\_split\_kwargs)  
1045 split\_dict.add(split\_generator.split\_info)  
1047 try:  
1048 # Prepare split will record examples associated to the split  
 → 1049 self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs)  
1050 except OSError as e:  
1051 raise OSError(  
1052 "Cannot find data file. "  
1053 + (self.manual\_download\_instructions or “”)  
1054 + “\nOriginal error:\n”  
1055 + str(e)  
1056 ) from None

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\datasets\builder.py:1813, in ArrowBasedBuilder.\_prepare\_split(self, split\_generator, file\_format, num\_proc, max\_shard\_size)  
1811 job\_id = 0  
1812 with pbar:  
 → 1813 for job\_id, done, content in self.\_prepare\_split\_single(  
1814 gen\_kwargs=gen\_kwargs, job\_id=job\_id, \*\*\_prepare\_split\_args  
1815 ):  
1816 if done:  
1817 result = content

File ~\anaconda3\envs\myenv\_nlp\lib\site-packages\datasets\builder.py:1958, in ArrowBasedBuilder.\_prepare\_split\_single(self, gen\_kwargs, fpath, file\_format, max\_shard\_size, job\_id)  
1956 if isinstance(e, SchemaInferenceError) and e. **context** is not None:  
1957 e = e. **context**  
 → 1958 raise DatasetGenerationError(“An error occurred while generating the dataset”) from e  
1960 yield job\_id, True, (total\_num\_examples, total\_num\_bytes, writer.\_features, num\_shards, shard\_lengths)

DatasetGenerationError: An error occurred while generating the dataset

---

<div class="post-metadata">

**Author:** ![Weibo021](https://avatars.discourse-cdn.com/v4/letter/w/0ea827/32.png) [@Weibo021](https://discuss.huggingface.co/u/Weibo021)\
**Post date:** [October 25, 2023, 11:34am UTC](https://discuss.huggingface.co/t/datasetgenerationerror-while-loading-dataset/58841/2 "2023-10-25T11:34:50Z")

</div>

I met same problem ：  
FileNotFoundError: [Errno 2] No such file or directory: 'C:/Users/Dell/.cache/huggingface/datasets/zeroshot\_\_\_twitter-financial-news-sentiment/default-9316297f35209a5d/0.0.0/eea64c71ca8b46dd3f537ed218fc9bf495d5707789152eb2764f5c78fa66d59d.incomplete/twitter-financial-news-sentiment-validation-00000-00000-of-NNNNN.arrow

I find there is no ‘default-9316297f35209a5d/0.0.0/eea64c71ca8b46dd3f537ed218fc9bf495d5707789152eb2764f5c78fa66d59d.incomplete’ directory under ‘'C:/Users/Dell/.cache/huggingface/datasets/zeroshot\_\_\_twitter-financial-news-sentiment/’ directory

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [October 26, 2023, 2:30pm UTC](https://discuss.huggingface.co/t/datasetgenerationerror-while-loading-dataset/58841/3 "2023-10-26T14:30:21Z")

</div>

Hmm, that’s weird. Does this error happen consistently?

---

<div class="post-metadata">

**Author:** ![xllcrn](https://avatars.discourse-cdn.com/v4/letter/x/4491bb/32.png) [@xllcrn](https://discuss.huggingface.co/u/xllcrn)\
**Post date:** [October 26, 2023, 2:53pm UTC](https://discuss.huggingface.co/t/datasetgenerationerror-while-loading-dataset/58841/4 "2023-10-26T14:53:00Z")

</div>

It works with the version 2.13.2 and not after.  
It works with the last version (2.14.6) with the parameter streaming=true of load\_dataset.  
It works on Colab (2.14.6) but not with jupyter notebook.  
Hopes it can help.