# Strange Error While Attempting to Load DataSet **URL:** \ **Category:** 🤗Datasets\ **Created:** [September 11, 2023, 8:01pm UTC](https://discuss.huggingface.co/t/strange-error-while-attempting-to-load-dataset/54531 "2023-09-11T20:01:05Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![FDSRashid](https://avatars.discourse-cdn.com/v4/letter/f/7ea924/32.png) [@FDSRashid](https://discuss.huggingface.co/u/FDSRashid)\ **Post date:** [September 11, 2023, 8:01pm UTC](https://discuss.huggingface.co/t/strange-error-while-attempting-to-load-dataset/54531/1 "2023-09-11T20:01:05Z")
Hi all, I’m kind of a beginner with the HF Interface, I was trying to load a 16 MB dataset with arabic characters, and I get the following error: I’m honestly confused what the error is. 0/site-packages/datasets/table.py", line 1833, in wrapper return func(array, \*args, \*\*kwargs) File “/home/user/.local/lib/python3.10/site-packages/datasets/table.py”, line 2027, in array\_cast return array.cast(pa\_type) File “pyarrow/array.pxi”, line 980, in pyarrow.lib.Array.cast File “/home/user/.local/lib/python3.10/site-packages/pyarrow/compute.py”, line 403, in cast return call\_function(“cast”, [arr], options, memory\_pool) File “pyarrow/\_compute.pyx”, line 572, in pyarrow.\_compute.call\_function File “pyarrow/\_compute.pyx”, line 367, in pyarrow.\_compute.Function.call File “pyarrow/error.pxi”, line 144, in pyarrow.lib.pyarrow\_internal\_check\_status File “pyarrow/error.pxi”, line 100, in pyarrow.lib.check\_status pyarrow.lib.ArrowInvalid: Failed to parse string: ‘17 - “”’ as a scalar of type int64 The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/home/user/app/app.py”, line 9, in dataset = load\_dataset(‘FDSRashid/hadith\_info’,data\_files = ‘Basic\_Edge\_Information.csv’, token = Secret\_token, split = ‘train’) File “/home/user/.local/lib/python3.10/site-packages/datasets/load.py”, line 2153, in load\_dataset builder\_instance.download\_and\_prepare( File “/home/user/.local/lib/python3.10/site-packages/datasets/builder.py”, line 954, in download\_and\_prepare self.\_download\_and\_prepare( File “/home/user/.local/lib/python3.10/site-packages/datasets/builder.py”, line 1049, in \_download\_and\_prepare self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs) File “/home/user/.local/lib/python3.10/site-packages/datasets/builder.py”, line 1813, in \_prepare\_split for job\_id, done, content in self.\_prepare\_split\_single( File “/home/user/.local/lib/python3.10/site-packages/datasets/builder.py”, line 1958, in \_prepare\_split\_single raise DatasetGenerationError(“An error occurred while generating the dataset”) from e datasets.builder.DatasetGenerationError: An error occurred while generating the dataset ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [September 21, 2023, 10:36am UTC](https://discuss.huggingface.co/t/strange-error-while-attempting-to-load-dataset/54531/2 "2023-09-21T10:36:53Z")
It looks like your dataset has data of incoherent types. There seems to be a column that is loaded as type “int64” but the dataset content `17 - “”` can’t be converted to an integer. Could you share some data samples and the code you used to load the dataset ? That would be helpful to investigate why you end up with this error ---
**Author:** ![FDSRashid](https://avatars.discourse-cdn.com/v4/letter/f/7ea924/32.png) [@FDSRashid](https://discuss.huggingface.co/u/FDSRashid)\ **Post date:** [September 24, 2023, 3:30am UTC](https://discuss.huggingface.co/t/strange-error-while-attempting-to-load-dataset/54531/3 "2023-09-24T03:30:44Z")
this was precisely the error! i simply loaded the dataset using `load_dataset('path/to/dataset')` , without any modification to the dataset. there were some invalid rows with their values and some null values in the dataset - pyarrow chose the default datatype to be integers. i made a temporary fix by making a column schema and setting the data type of all the columns to string. however this leads me to my second issue, loading in datasets with null values. even when i set the column type to all be string, null values aren’t read in and load\_dataset yields an error . now i’m confused on how to read in datasets with null values using the load\_datasets() function. 😆 ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [September 24, 2023, 3:39pm UTC](https://discuss.huggingface.co/t/strange-error-while-attempting-to-load-dataset/54531/4 "2023-09-24T15:39:16Z")
What’s the error messager ? `load_dataset` should work even if you have null values ---
**Author:** ![FDSRashid](https://avatars.discourse-cdn.com/v4/letter/f/7ea924/32.png) [@FDSRashid](https://discuss.huggingface.co/u/FDSRashid)\ **Post date:** [September 24, 2023, 5:40pm UTC](https://discuss.huggingface.co/t/strange-error-while-attempting-to-load-dataset/54531/5 "2023-09-24T17:40:39Z")
this is my column schema : `features = Features({'Book_ID': Value('int32'),'taraf_ID': Value('string'), 'Hadith_ID': Value('string'), 'matn': Value('string'), 'taraf_tally': Value('int32'), 'wordcount': Value('string'), 'Domain': Value('string'), 'Category': Value('string'), 'translation': Value('string')})` . When i try to to load in this dataset using this code : `dataset = load_dataset("FDSRashid/hadith_info", data_files = 'All_Matns.csv', token = string1, features = features)`, i get the following error : ```auto Failed to read file '/root/.cache/huggingface/datasets/downloads/ac7e243c60b61b8decc6fc884b4b76a7d6c12164953ec0f10a672362460a1bcd' with error : cannot safely convert passed user dtype of int32 for object dtyped data in column 4 ERROR:datasets.packaged_modules.csv.csv:Failed to read file '/root/.cache/huggingface/datasets/downloads/ac7e243c60b61b8decc6fc884b4b76a7d6c12164953ec0f10a672362460a1bcd' with error : cannot safely convert passed user dtype of int32 for object dtyped data in column 4 --------------------------------------------------------------------------- TypeError Traceback (most recent call last) /usr/local/lib/python3.10/dist-packages/pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_tokens() TypeError: Cannot cast array data from dtype('O') to dtype('int32') according to the rule 'safe' During handling of the above exception, another exception occurred: ValueError Traceback (most recent call last) 15 frames ValueError: cannot safely convert passed user dtype of int32 for object dtyped data in column 4 The above exception was the direct cause of the following exception: DatasetGenerationError Traceback (most recent call last) /usr/local/lib/python3.10/dist-packages/datasets/builder.py in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id) 1956 if isinstance(e, SchemaInferenceError) and e. __context__ is not None: 1957 e = e. __context__ -> 1958 raise DatasetGenerationError("An error occurred while generating the dataset") from e 1959 1960 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths) DatasetGenerationError: An error occurred while generating the dataset ``` I did successfully load everything when it was a string, apologies for the confusion. but if i have numeric data with some empty values, is the only way to load them by passing them as string? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [September 25, 2023, 9:33am UTC](https://discuss.huggingface.co/t/strange-error-while-attempting-to-load-dataset/54531/6 "2023-09-25T09:33:08Z")
Integers in CSV can be loaded as integers in general. However in your case your CSV contains integers formatted like “1\_000” instead of “1000” for example, and pandas doesn’t support it ---
**Author:** ![numpy01](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/numpy01/32/44292_2.png) [@numpy01](https://discuss.huggingface.co/u/numpy01)\ **Post date:** [March 28, 2025, 1:39pm UTC](https://discuss.huggingface.co/t/strange-error-while-attempting-to-load-dataset/54531/7 "2025-03-28T13:39:37Z")
## good day here. i don’t konw if i can ask my questions on this discussion, cause i can’t find where to ask my own question. this is the error am facing ValueError Traceback (most recent call last) Cell In[127], line 95 89 callbacks = [ 90 EarlyStopping(monitor=‘val\_loss’, patience=5, restore\_best\_weights=True), 91 ReduceLROnPlateau(monitor=‘val\_loss’, factor=0.5, patience=3, verbose=1) 92 ] 94 # Train model —\> 95 history = model.fit( 96 train\_generator, 97 validation\_data=val\_generator, 98 epochs=20, 99 callbacks=callbacks 100 ) 102 # Plot training curves 103 plt.figure(figsize=(10, 5)) File ~\anaconda3\Lib\site-packages\keras\src\utils\traceback\_utils.py:122, in filter\_traceback..error\_handler(\*args, \*\*kwargs) 119 filtered\_tb = \_process\_traceback\_frames(e. **traceback** ) 120 # To get the full stack trace, call: 121 # `keras.config.disable_traceback_filtering()` → 122 raise e.with\_traceback(filtered\_tb) from None 123 finally: 124 del filtered\_tb File ~\anaconda3\Lib\site-packages\keras\src\trainers\data\_adapters\py\_dataset\_adapter.py:295, in PyDatasetAdapter.get\_tf\_dataset(self) 290 batches = [ 291 self.\_standardize\_batch(self.py\_dataset[i]) 292 for i in range(num\_samples) 293 ] 294 if len(batches) == 0: → 295 raise ValueError(“The PyDataset has length 0”) 296 self.\_output\_signature = data\_adapter\_utils.get\_tensor\_spec(batches) 298 ds = tf.data.Dataset.from\_generator( 299 self.\_get\_iterator, 300 output\_signature=self.\_output\_signature, 301 ) ValueError: The PyDataset has length 0 ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [March 28, 2025, 2:55pm UTC](https://discuss.huggingface.co/t/strange-error-while-attempting-to-load-dataset/54531/8 "2025-03-28T14:55:38Z")
TensorFlow error? > > > \### System information > \- \*\*Have I written custom code (as opposed to using a st…ock example script provided in TensorFlow)\*\*: Custom > \- \*\*OS Platform and Distribution (e.g., Linux Ubuntu 16.04)\*\*: Windows 10 > \- \*\*TensorFlow installed from (source or binary)\*\*: Binary > \- \*\*TensorFlow version (use command below)\*\*: 1.9-rc0 > \- \*\*Python version\*\*: 3.6 > \- \*\*Bazel version (if compiling from source)\*\*: NA > \- \*\*GCC/Compiler version (if compiling from source)\*\*: NA > \- \*\*CUDA/cuDNN version\*\*: NA > \- \*\*GPU model and memory\*\*: NA > \- \*\*Exact command to reproduce\*\*: \`model.fit(get\_iterator,steps\_per\_epoch=2,batch\_size=2,epochs=2,shuffle =True,verbose=1)\` > and > \`model.fit(get\_iterator,get\_iterator,steps\_per\_epoch=2,batch\_size=2,epochs=2,shuffle =True,verbose=1)\` > > \### Describe the problem > When I pass one dataset iterator to \`fit\` method, I get: > > > \> Please provide data as a list or tuple of 2 elements - input and target pair. Received Tensor("IteratorGetNext\_4:0", shape=(2, ?), dtype=float32) > > > When I pass two iterators I get the error: > > \> ValueError: You passed a dataset or dataset iterator (\) as input \`x\` to your model. In that case, you should not specify a target (\`y\`) argument, since the dataset or dataset iterator generates both input data and target data. Received: \ > > When I create a new dataset after zipping the original x and y data set and pass that to \`fit \`I get the error described in https://github.com/tensorflow/tensorflow/issues/19912 > > According to 1.9-rc0 method release notes iterators should be usable with keras training methods. Please provide a solution or provide clarification in the documentation. > > \### Source code / logs > \`\`\` > dataset= tf.contrib.data.make\_csv\_dataset(file\_name,48,select\_columns= \['Load\_residential\_multi\_0','Load\_residential\_multi\_1'\],shuffle=False) > dataset = dataset.map(lambda x: tf.stack(list(x.values()))) > get\_iterator = dataset.make\_one\_shot\_iterator() > get\_batch = get\_iterator.get\_next() > > \#Building and training a single layer model using Keras (Available within TensorFlow) > model = Sequential() > \#Input Layer > model.add(InputLayer(input\_shape=(48,),name='InputLayer'))#,input\_tensor =dataset > \#model.add(BatchNormalization(axis=1)) #Normalizing values > \#Layer1 > model.add(Dense(units=5,activation='relu',name='FeedForward1')) #Add a feed forward layer > \#Layer2 > model.add(Dense(units=5,activation='relu',name='FeedForward2')) #Add a feed forward layer > \#Output layer > model.add(Dense(units=48,name='OutputLayer')) > > \#Specify los function and optimizer > model.compile(loss='mse',optimizer='adam',metrics=\['mae'\]) > > \#Summarize model > model.summary() > \#Train the model > model.fit(get\_iterator,steps\_per\_epoch=2,batch\_size=2,epochs=2,shuffle =True,verbose=1) > \#model.fit(get\_iterator,get\_iterator,steps\_per\_epoch=2,batch\_size=2,epochs=2,shuffle =True,verbose=1) > \`\`\` > > > > > \*\*Keras Version:\*\* 3.5.0 > \*\*Tensorflow Version:\*\* 2.17.0 > > \*\*What I want to do:…\*\* > Use PyDataset class in a data distributed environment. > > \--- > > I would like to ask about the status of PyDataset and some of its best uses and practices. I have a functioning PyDataset class that ingests and processes HDF files: > > \`\`\` > class HDFDataset(K.utils.PyDataset): > ''' > Keras data loader to replace Tensorflow's Dataset API. > Reads HDF5 files. > Inputs: > file\_list: list > list of file names, pre-globbed > batch\_size: int > size of batches > shuffle: bool > whether or not to shuffle the dataset at the end of each epoch > lons\_lats: bool > whether or not to include longitudes and latitudes > > -- Additional keyword arguments -- > workers=1 > use\_multiprocessing=False > max\_queue\_size=10 > ''' > def \_\_init\_\_(self, > file\_list: list | tuple | set, > batch\_size: int, > shuffle: bool = False, > lons\_lats: bool = False, > subsample: bool = False, > \*\*kwargs): > super(HDFDataset, self).\_\_init\_\_(\*\*kwargs) > self.shuffle = shuffle > self.batch\_size = batch\_size > self.tmplen = len(self.file\_list) > > self.subsample = subsample > if self.subsample: > self.slice = slice(64, 192) > self.time\_slice = slice(0,6) > > def \_\_len\_\_(self): > return self.tmplen // self.batch\_size > > def \_extract\_data\_from\_hdf5(self, file\_list): > input\_list = \['priors', 'model'\] > output\_list = \['forecast'\] > > # Preparing input dictionary > inputs\_dict = {} > for name in input\_list: > new\_var = f'input\_{name}' > inputs\_dict\[new\_var\] = \[\] > > outputs = \[\] > > for f in file\_list: > with h5py.File(f, 'r') as h5: > for k in input\_list: > new\_var = f'input\_{k}' > if self.subsample: > inputs\_dict\[new\_var\].append(h5.get(k)\[:, self.slice, self.slice, :\]) > else: > inputs\_dict\[new\_var\].append(h5.get(k)\[...\]) > for k in output\_list: > if self.subsample: > outputs.append(h5.get(k)\[0:6, self.slice, self.slice, :\]) > else: > outputs.append(h5.get(k)\[0:6, self.slice, self.slice, :\]) > > for k in input\_list: > nv = f'input\_{k}' > inputs\_dict\[nv\] = np.stack(inputs\_dict\[nv\], axis = 0) > > outputs = np.stack(outputs, axis = 0) > > return inputs\_dict, outputs > > def \_\_getitem\_\_(self, > idx: int): > > if idx \>= self.\_\_len\_\_(): raise StopIteration > > low = idx \* self.batch\_size > > high = min(low + self.batch\_size, self.tmplen) > > inputs, outputs = self.\_extract\_data\_from\_hdf5(self.file\_list\[low:high\]) > > return \[inputs, outputs\] > > def on\_epoch\_end(self): > if self.shuffle: random.shuffle(self.file\_list) # In-place shuffle > \`\`\` > > This works for my case really nicely. It avoids the memory leak nightmare with which I have been dealing by directly trying to use the \`tf.data\` API (https://github.com/tensorflow/tensorflow/issues/72014) for multiple inputs from the same file. > > But the documentation on PyDataset stinks! > > Looking inside the \[source code\](https://github.com/keras-team/keras/blob/v3.5.0/keras/src/trainers/data\_adapters/py\_dataset\_adapter.py), PyDataset has an Adapter class that will make a Tensorflow data generator. Does this automatically get called during \`fit()\`? Is it best practice to call the data generator directly so I can distribute the dataset via \[TF's experimental distribute dataset function\](https://www.tensorflow.org/api\_docs/python/tf/distribute/Strategy#experimental\_distribute\_dataset)? > > In the source, there is also a \`PyDatasetEnqueuer\` class. Do I need this? Why is this here? Who is the target audience? Is the expectation of the Enquerer in the PyDataset class also the reason I need to raise a \`StopIteration\` command in \`\_\_getitem\_\_\`? > > Also digging into source, at \[this point\](https://github.com/keras-team/keras/blob/v3.5.0/keras/src/trainers/data\_adapters/py\_dataset\_adapter.py), the shuffle is hard-coded to 8. That probably needs to go. > > Anyway, I don't have any specific programming questions here, but I would like to know what best practices are, how do I use \`PyDataset\` in a (Tensorflow) distributed data environment, and so on.