# Error converting np float32

**URL:** <https://discuss.huggingface.co/t/error-converting-np-float32/165714>\
**Category:** 🤗Datasets\
**Created:** [August 5, 2025, 10:48am UTC](https://discuss.huggingface.co/t/error-converting-np-float32/165714 "2025-08-05T10:48:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sandhyavs](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sandhyavs](https://discuss.huggingface.co/u/sandhyavs)\
**Post date:** [August 5, 2025, 10:48am UTC](https://discuss.huggingface.co/t/error-converting-np-float32/165714/1 "2025-08-05T10:48:27Z")

</div>

Setup: Nvidia Jetson, using ROS2 to collect data. When I collect data on my local system over ssh, it works fine, on the Jetson I get this issue:

```auto
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert 0.037997484 with type numpy.float32: did not recognize Python value type when inferring an Arrow data type

```

Versions on Jetson:

```auto
On my jetson:
(lerobot) root@dusty-1000052:/home/dusty/lerobot/lerobot-clutterbot# pip freeze | grep -E 'pyarrow|datasets|numpy|protobuf'
datasets==3.6.0
numpy @ file:///croot/numpy_and_numpy_base_1672336188316/work
protobuf==6.31.1
pyarrow==20.0.0
(lerobot) root@dusty-1000052:/home/dusty/lerobot/lerobot-clutterbot# python -c "import numpy; print(numpy. __version__ )"
1.23.5

```

Detailed error log:

```auto
    response = fn(cfg, *args, **kwargs)
  File "/home/dusty/lerobot/lerobot/src/lerobot/record.py", line 380, in record
    dataset.save_episode()
  File "/home/dusty/lerobot/lerobot/src/lerobot/datasets/lerobot_dataset.py", line 850, in save_episode
    self._save_episode_table(episode_buffer, episode_index)
  File "/home/dusty/lerobot/lerobot/src/lerobot/datasets/lerobot_dataset.py", line 896, in _save_episode_table
    ep_dataset = datasets.Dataset.from_dict(episode_dict, features=self.hf_features, split="train")
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 940, in from_dict
    pa_table = InMemoryTable.from_pydict(mapping=mapping)
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/datasets/table.py", line 758, in from_pydict
    return cls(pa.Table.from_pydict(*args, **kwargs))
  File "pyarrow/table.pxi", line 1982, in pyarrow.lib._Tabular.from_pydict
  File "pyarrow/table.pxi", line 6379, in pyarrow.lib._from_pydict
  File "pyarrow/array.pxi", line 405, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 255, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 117, in pyarrow.lib._handle_arrow_array_protocol
  File "/root/miniconda3/envs/lerobot/lib/python3.10/site-packages/datasets/arrow_writer.py", line 311, in __arrow_array__
    out = pa.array(cast_to_python_objects(data, only_1d_for_numpy=True, optimize_list_casting=False))
  File "pyarrow/array.pxi", line 375, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 45, in pyarrow.lib._sequence_to_array
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not convert 0.037997484 with type numpy.float32: did not recognize Python value type when inferring an Arrow data type

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 5, 2025, 11:13am UTC](https://discuss.huggingface.co/t/error-converting-np-float32/165714/2 "2025-08-05T11:13:35Z")

</div>

> datasets==3.6.0

I think the culprit is probably this. The old datasets library and old arrow library dug up ancient issues. If it is possible to upgrade the library, that would be the quickest solution, but if not, try using an old workaround.

> <https://github.com/huggingface/datasets/issues/6007>
>
> \### Describe the bug
> 
> When load a large dataset with the following code
> 
> \`\`\`…python
> from datasets import load\_dataset
> dataset = load\_dataset("liwu/MNBVC", 'news\_peoples\_daily', split='train')
> \`\`\`
> 
> We encountered the error: "OverflowError: Python int too large to convert to C long"
> The error look something like:
> 
> \`\`\`
> OverflowError: Python int too large to convert to C long
> 
> During handling of the above exception, another exception occurred:
> 
> OverflowError Traceback (most recent call last)
> \<ipython-input-7-0ed8700e662d\> in \<module\>
> \----\> 1 dataset = load\_dataset("liwu/MNBVC", 'news\_peoples\_daily', split='train', cache\_dir='/sfs/MNBVC/.cache/')
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/load.py in load\_dataset(path, name, data\_dir, data\_files, split, cache\_dir, features, download\_config, download\_mode, ignore\_verifications, keep\_in\_memory, save\_infos, revision, use\_auth\_token, task, streaming, \*\*config\_kwargs)
> 1749 ignore\_verifications=ignore\_verifications,
> 1750 try\_from\_hf\_gcs=try\_from\_hf\_gcs,
> \-\> 1751 use\_auth\_token=use\_auth\_token,
> 1752 )
> 1753 
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in download\_and\_prepare(self, download\_config, download\_mode, ignore\_verifications, try\_from\_hf\_gcs, dl\_manager, base\_path, use\_auth\_token, \*\*download\_and\_prepare\_kwargs)
> 703 if not downloaded\_from\_gcs:
> 704 self.\_download\_and\_prepare(
> \--\> 705 dl\_manager=dl\_manager, verify\_infos=verify\_infos, \*\*download\_and\_prepare\_kwargs
> 706 )
> 707 # Sync info
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in \_download\_and\_prepare(self, dl\_manager, verify\_infos)
> 1225 
> 1226 def \_download\_and\_prepare(self, dl\_manager, verify\_infos):
> \-\> 1227 super().\_download\_and\_prepare(dl\_manager, verify\_infos, check\_duplicate\_keys=verify\_infos)
> 1228 
> 1229 def \_get\_examples\_iterable\_for\_split(self, split\_generator: SplitGenerator) -\> ExamplesIterable:
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in \_download\_and\_prepare(self, dl\_manager, verify\_infos, \*\*prepare\_split\_kwargs)
> 791 try:
> 792 # Prepare split will record examples associated to the split
> \--\> 793 self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs)
> 794 except OSError as e:
> 795 raise OSError(
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/builder.py in \_prepare\_split(self, split\_generator, check\_duplicate\_keys)
> 1219 writer.write(example, key)
> 1220 finally:
> \-\> 1221 num\_examples, num\_bytes = writer.finalize()
> 1222 
> 1223 split\_generator.split\_info.num\_examples = num\_examples
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow\_writer.py in finalize(self, close\_stream)
> 536 # Re-intializing to empty list for next batch
> 537 self.hkey\_record = \[\]
> \--\> 538 self.write\_examples\_on\_file()
> 539 if self.pa\_writer is None:
> 540 if self.schema:
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow\_writer.py in write\_examples\_on\_file(self)
> 407 # Since current\_examples contains (example, key) tuples
> 408 batch\_examples\[col\] = \[row\[0\]\[col\] for row in self.current\_examples\]
> \--\> 409 self.write\_batch(batch\_examples=batch\_examples)
> 410 self.current\_examples = \[\]
> 411 
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow\_writer.py in write\_batch(self, batch\_examples, writer\_batch\_size)
> 506 col\_try\_type = try\_features\[col\] if try\_features is not None and col in try\_features else None
> 507 typed\_sequence = OptimizedTypedSequence(batch\_examples\[col\], type=col\_type, try\_type=col\_try\_type, col=col)
> \--\> 508 arrays.append(pa.array(typed\_sequence))
> 509 inferred\_features\[col\] = typed\_sequence.get\_inferred\_type()
> 510 schema = inferred\_features.arrow\_schema if self.pa\_writer is None else self.schema
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.array()
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.\_handle\_arrow\_array\_protocol()
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/datasets/arrow\_writer.py in \_\_arrow\_array\_\_(self, type)
> 180 else:
> 181 trying\_cast\_to\_python\_objects = True
> \--\> 182 out = pa.array(cast\_to\_python\_objects(data, only\_1d\_for\_numpy=True))
> 183 # use smaller integer precisions if possible
> 184 if self.trying\_int\_optimization:
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.array()
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/array.pxi in pyarrow.lib.\_sequence\_to\_array()
> 
> /sfs/MNBVC/venv/lib64/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow\_internal\_check\_status()
> 
> OverflowError: Python int too large to convert to C long
> \`\`\`
> 
> However, that dataset can be loaded in a streaming manner:
> 
> \`\`\`python
> from datasets import load\_dataset
> dataset = load\_dataset("liwu/MNBVC", 'news\_peoples\_daily', split='train', streaming=True)
> 
> for i in dataset:
> pass # it work well
> \`\`\`
> 
> Another issue is reported in our dataset hub:
> https://huggingface.co/datasets/liwu/MNBVC/discussions/2
> 
> 
> \### Steps to reproduce the bug
> 
> from datasets import load\_dataset
> dataset = load\_dataset("liwu/MNBVC", 'news\_peoples\_daily', split='train')
> 
> \### Expected behavior
> 
> the dataset can be safely loaded
> 
> \### Environment info
> 
> \- \`datasets\` version: 2.4.0
> \- Platform: Linux-3.10.0-1160.an7.x86\_64-x86\_64-with-centos-7.9
> \- Python version: 3.6.8
> \- PyArrow version: 6.0.1
> \- Pandas version: 1.1.5

> <https://stackoverflow.com/questions/59636745/pyarrow-lib-arrowinvalid-could-not-convert-x-with-type-y-did-not-recognize-p>

> [@Setting dataset feature value as numpy array](https://discuss.huggingface.co/t/setting-dataset-feature-value-as-numpy-array/20940):
>
> Python: 3.9.7 Datsets: 2.1.0 I have a dataset where each example has a label and array-like sequence of floats associated with it. The dataset is very large and I have opted to create a loading script following [these instructions](https://huggingface.co/docs/datasets/dataset_script#testing-data-and-checksum-metadata). Specifically, my data looks something like this: label | data -------------------------- 'label\_1' | [-3.05175781e-05, 3.35693359e-04, -2.62451172e-03, 2.44140625e-03, ...] 'label\_7' | [...] . . . where for each example the data column is a numpy array. When b…

---

<div class="post-metadata">

**Author:** ![sandhyavs](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@sandhyavs](https://discuss.huggingface.co/u/sandhyavs)\
**Post date:** [August 5, 2025, 11:49am UTC](https://discuss.huggingface.co/t/error-converting-np-float32/165714/4 "2025-08-05T11:49:41Z")

</div>

Thanks for the reply, I tried upgrading but it still doesnt work. I’m also running it with the Lerobot library which has the requirement of datasets being 3.6, I’m still getting the same issue so I guess 4.0 is still compatible.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [August 5, 2025, 10:48pm UTC](https://discuss.huggingface.co/t/error-converting-np-float32/165714/5 "2025-08-05T22:48:39Z")

</div>

Hmm, maybe [try updating PyArrow](https://github.com/apache/arrow/issues/9462) together?  
It’s also used in the `datasets` library.

```bash
pip install -U pyarrow>=21.0.0 datasets

```