# Error converting np float32

**URL:** <https://discuss.huggingface.co/t/error-converting-np-float32/165714>\
**Category:** 🤗Datasets\
**Created:** [August 5, 2025, 10:48am UTC](https://discuss.huggingface.co/t/error-converting-np-float32/165714 "2025-08-05T10:48:27Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 5, 2025, 11:13am UTC](https://discuss.huggingface.co/t/error-converting-np-float32/165714/2 "2025-08-05T11:13:35Z")

</div>

> datasets==3.6.0

I think the culprit is probably this. The old datasets library and old arrow library dug up ancient issues. If it is possible to upgrade the library, that would be the quickest solution, but if not, try using an old workaround.

> <https://github.com/huggingface/datasets/issues/6007>
>
> \### Describe the bug
> 
> When load a large dataset with the following code
> 
> \`\`\`…python
> from datasets import load\_dataset
> dataset = load\_dataset("liwu/MNBVC", 'news\_peoples\_daily', split='train')
> \`\`\`
> 
> We encountered the error: "OverflowError: Python int too large to convert to C long"
> The error look something like:
> 
> \`\`\`
> OverflowError: Python int too large to convert to C long
> 
> During handling of the above exception, another exception occurred:
> 
> OverflowError Traceback (most recent call last)
> \<ipython-input-7-0ed8700e662d\> in \<module\>
> \----\> 1 dataset = load\_dataset("liwu/MNBVC", 'news\_peoples\_daily', split='train', cache\_dir='/sfs/MNBVC/.cache/')
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/load.py in load\_dataset(path, name, data\_dir, data\_files, split, cache\_dir, features, download\_config, download\_mode, ignore\_verifications, keep\_in\_memory, save\_infos, revision, use\_auth\_token, task, streaming, \*\*config\_kwargs)
> 1749 ignore\_verifications=ignore\_verifications,
> 1750 try\_from\_hf\_gcs=try\_from\_hf\_gcs,
> \-\> 1751 use\_auth\_token=use\_auth\_token,
> 1752 )
> 1753 
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in download\_and\_prepare(self, download\_config, download\_mode, ignore\_verifications, try\_from\_hf\_gcs, dl\_manager, base\_path, use\_auth\_token, \*\*download\_and\_prepare\_kwargs)
> 703 if not downloaded\_from\_gcs:
> 704 self.\_download\_and\_prepare(
> \--\> 705 dl\_manager=dl\_manager, verify\_infos=verify\_infos, \*\*download\_and\_prepare\_kwargs
> 706 )
> 707 # Sync info
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in \_download\_and\_prepare(self, dl\_manager, verify\_infos)
> 1225 
> 1226 def \_download\_and\_prepare(self, dl\_manager, verify\_infos):
> \-\> 1227 super().\_download\_and\_prepare(dl\_manager, verify\_infos, check\_duplicate\_keys=verify\_infos)
> 1228 
> 1229 def \_get\_examples\_iterable\_for\_split(self, split\_generator: SplitGenerator) -\> ExamplesIterable:
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in \_download\_and\_prepare(self, dl\_manager, verify\_infos, \*\*prepare\_split\_kwargs)
> 791 try:
> 792 # Prepare split will record examples associated to the split
> \--\> 793 self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs)
> 794 except OSError as e:
> 795 raise OSError(
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in \_prepare\_split(self, split\_generator, check\_duplicate\_keys)
> 1219 writer.write(example, key)
> 1220 finally:
> \-\> 1221 num\_examples, num\_bytes = writer.finalize()
> 1222 
> 1223 split\_generator.split\_info.num\_examples = num\_examples
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow\_writer.py in finalize(self, close\_stream)
> 536 # Re-intializing to empty list for next batch
> 537 self.hkey\_record = \[\]
> \--\> 538 self.write\_examples\_on\_file()
> 539 if self.pa\_writer is None:
> 540 if self.schema:
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow\_writer.py in write\_examples\_on\_file(self)
> 407 # Since current\_examples contains (example, key) tuples
> 408 batch\_examples\[col\] = \[row\[0\]\[col\] for row in self.current\_examples\]
> \--\> 409 self.write\_batch(batch\_examples=batch\_examples)
> 410 self.current\_examples = \[\]
> 411 
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow\_writer.py in write\_batch(self, batch\_examples, writer\_batch\_size)
> 506 col\_try\_type = try\_features\[col\] if try\_features is not None and col in try\_features else None
> 507 typed\_sequence = OptimizedTypedSequence(batch\_examples\[col\], type=col\_type, try\_type=col\_try\_type, col=col)
> \--\> 508 arrays.append(pa.array(typed\_sequence))
> 509 inferred\_features\[col\] = typed\_sequence.get\_inferred\_type()
> 510 schema = inferred\_features.arrow\_schema if self.pa\_writer is None else self.schema
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.array()
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.\_handle\_arrow\_array\_protocol()
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow\_writer.py in \_\_arrow\_array\_\_(self, type)
> 180 else:
> 181 trying\_cast\_to\_python\_objects = True
> \--\> 182 out = pa.array(cast\_to\_python\_objects(data, only\_1d\_for\_numpy=True))
> 183 # use smaller integer precisions if possible
> 184 if self.trying\_int\_optimization:
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.array()
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.\_sequence\_to\_array()
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow\_internal\_check\_status()
> 
> OverflowError: Python int too large to convert to C long
> \`\`\`
> 
> However, that dataset can be loaded in a streaming manner:
> 
> \`\`\`python
> from datasets import load\_dataset
> dataset = load\_dataset("liwu/MNBVC", 'news\_peoples\_daily', split='train', streaming=True)
> 
> for i in dataset:
> pass # it work well
> \`\`\`
> 
> Another issue is reported in our dataset hub:
> https://huggingface.co/datasets/liwu/MNBVC/discussions/2
> 
> 
> \### Steps to reproduce the bug
> 
> from datasets import load\_dataset
> dataset = load\_dataset("liwu/MNBVC", 'news\_peoples\_daily', split='train')
> 
> \### Expected behavior
> 
> the dataset can be safely loaded
> 
> \### Environment info
> 
> \- \`datasets\` version: 2.4.0
> \- Platform: Linux-3.10.0-1160.an7.x86\_64-x86\_64-with-centos-7.9
> \- Python version: 3.6.8
> \- PyArrow version: 6.0.1
> \- Pandas version: 1.1.5

> <https://stackoverflow.com/questions/59636745/pyarrow-lib-arrowinvalid-could-not-convert-x-with-type-y-did-not-recognize-p>

> [@Setting dataset feature value as numpy array](https://discuss.huggingface.co/t/setting-dataset-feature-value-as-numpy-array/20940):
>
> Python: 3.9.7 Datsets: 2.1.0 I have a dataset where each example has a label and array-like sequence of floats associated with it. The dataset is very large and I have opted to create a loading script following [these instructions](https://huggingface.co/docs/datasets/dataset_script#testing-data-and-checksum-metadata). Specifically, my data looks something like this: label | data -------------------------- 'label\_1' | [-3.05175781e-05, 3.35693359e-04, -2.62451172e-03, 2.44140625e-03, ...] 'label\_7' | [...] . . . where for each example the data column is a numpy array. When b…

---

_[View the full topic](https://discuss.huggingface.co/t/error-converting-np-float32/165714)._