# pyarrow.lib.FloatArray: did not recognize Python value type when inferring an Arrow data type

**URL:** <https://discuss.huggingface.co/t/pyarrow-lib-floatarray-did-not-recognize-python-value-type-when-inferring-an-arrow-data-type/33417>\
**Category:** 🤗Datasets\
**Created:** [March 9, 2023, 11:00am UTC](https://discuss.huggingface.co/t/pyarrow-lib-floatarray-did-not-recognize-python-value-type-when-inferring-an-arrow-data-type/33417 "2023-03-09T11:00:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![MarLac](https://avatars.discourse-cdn.com/v4/letter/m/bbce88/32.png) [@MarLac](https://discuss.huggingface.co/u/MarLac)\
**Post date:** [March 9, 2023, 11:00am UTC](https://discuss.huggingface.co/t/pyarrow-lib-floatarray-did-not-recognize-python-value-type-when-inferring-an-arrow-data-type/33417/1 "2023-03-09T11:00:30Z")

</div>

Hi everyone!

I work with a large dataset that I want to convert into a Huggingface dataset.  
As my workspace and the dataset workspace are not on the same device, I have created a HDF5 file (with h5py) that I have transmitted on my workspace.

Now I want to open that file and give the data to an empty dataset.  
My code is the following:

```auto
import h5py
import datasets
import pyarrow as py

# Load the HDF5 file
SAVE_DIR = './data/'

features = h5py.File(SAVE_DIR+'features.hdf5','r')

# Validation
print("Process validation data")
valid_data = features["validation"]["data/features"]

print("Extract validation values")
v_array_values = [np.float32(item[()]) for item in valid_data.values()]

print("Create validation dataset")
v_array = [py.array(item, type=py.float32()) for item in v_array_values]
dict_valid = datasets.Dataset.from_dict({'input_values': v_array})
dict_valid.save_to_disk(SAVE_DIR+"validation_dataset")

```

I know v\_array\_values is useless in this case, it is just for testing and ensuring that I have Float32 data, which is the original data type.  
However, I have this error:

> Traceback (most recent call last):  
> File “”, line 1, in   
> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\site-packages\datasets\arrow\_dataset.py”, line 859, in from\_dict  
> pa\_table = InMemoryTable.from\_pydict(mapping=mapping)  
> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\site-packages\datasets\table.py”, line 750, in from\_pydict  
> return cls(pa.Table.from\_pydict(\*args, \*\*kwargs))  
> File “pyarrow\table.pxi”, line 3625, in pyarrow.lib.Table.from\_pydict  
> File “pyarrow\table.pxi”, line 5150, in pyarrow.lib.\_from\_pydict  
> File “pyarrow\array.pxi”, line 342, in pyarrow.lib.asarray  
> File “pyarrow\array.pxi”, line 230, in pyarrow.lib.array  
> File “pyarrow\array.pxi”, line 110, in pyarrow.lib.\_handle\_arrow\_array\_protocol  
> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\site-packages\datasets\arrow\_writer.py”, line 239, in **arrow\_array**  
> out = pa.array(cast\_to\_python\_objects(data, only\_1d\_for\_numpy=True, optimize\_list\_casting=False))  
> File “pyarrow\array.pxi”, line 316, in pyarrow.lib.array  
> File “pyarrow\array.pxi”, line 39, in pyarrow.lib.\_sequence\_to\_array  
> File “pyarrow\error.pxi”, line 144, in pyarrow.lib.pyarrow\_internal\_check\_status  
> File “pyarrow\error.pxi”, line 100, in pyarrow.lib.check\_status  
> pyarrow.lib.ArrowInvalid: Could not convert \<pyarrow.lib.FloatArray object at 0x000002A3D769B760\>  
> [  
> -0.0000020123275,  
> -0.0000020123275,  
> -0.0000020123275,  
> -0.0000020123275,  
> -0.0000020123275,  
> -0.0000020123275,  
> -0.0000020123275,  
> -0.0000020123275,  
> -0.0000020123275,  
> -0.0000020123275,  
> …  
> 0.079432674,  
> -0.35756943,  
> -0.42882702,  
> 0.0595785,  
> 0.38494512,  
> 0.25881445,  
> 0.02550633,  
> -0.05002562,  
> -0.02196826,  
> -0.020529125  
> ] with type pyarrow.lib.FloatArray: did not recognize Python value type when inferring an Arrow data type

I tried a lot of stuff, like not converting my data in pyarrow array, using pandas, creating a dictionnary before, etc.  
I think maybe the problem could come from the HDF5 file but I cannot reproduce the system without it.  
Have you any idea of what that could come from?

Thank you!

---

<div class="post-metadata">

**Author:** ![MarLac](https://avatars.discourse-cdn.com/v4/letter/m/bbce88/32.png) [@MarLac](https://discuss.huggingface.co/u/MarLac)\
**Post date:** [March 10, 2023, 3:18pm UTC](https://discuss.huggingface.co/t/pyarrow-lib-floatarray-did-not-recognize-python-value-type-when-inferring-an-arrow-data-type/33417/2 "2023-03-10T15:18:34Z")

</div>

This error was because of v\_array and the use of py.array().  
However, when I remove that, I still have an error:

> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\runpy.py”, line 194, in \_run\_module\_as\_main  
> return \_run\_code(code, main\_globals, None,  
> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\runpy.py”, line 87, in _run\_code  
> exec(code, run\_globals)  
> File "c:\Users\<username\>.vscode\extensions\ms-python.python-2023.4.1\pythonFiles\lib\python\debugpy\adapter/…/…\debugpy\launcher/…/…\debugpy\_main_.py", line 39, in   
> cli.main()  
> File “c:\Users\<username\>.vscode\extensions\ms-python.python-2023.4.1\pythonFiles\lib\python\debugpy\adapter/…/…\debugpy\launcher/…/…\debugpy/…\debugpy\server\cli.py”, line 430, in main  
> run()  
> File “c:\Users\<username\>.vscode\extensions\ms-python.python-2023.4.1\pythonFiles\lib\python\debugpy\adapter/…/…\debugpy\launcher/…/…\debugpy/…\debugpy\server\cli.py”, line 284, in run\_file  
> runpy.run\_path(target, run\_name=“ **main** ”)  
> File “c:\Users\<username\>.vscode\extensions\ms-python.python-2023.4.1\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd\_bundle\pydevd\_runpy.py”, line 321, in run\_path  
> return \_run\_module\_code(code, init\_globals, run\_name,  
> File “c:\Users\<username\>.vscode\extensions\ms-python.python-2023.4.1\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd\_bundle\pydevd\_runpy.py”, line 135, in \_run\_module\_code  
> \_run\_code(code, mod\_globals, init\_globals,  
> File “c:\Users\<username\>.vscode\extensions\ms-python.python-2023.4.1\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd\_bundle\pydevd\_runpy.py”, line 124, in \_run\_code  
> exec(code, run\_globals)  
> File “c:\Users\<username\>\Documents\Codes\_Python\pretraining\_wav2vec\Code\_Lightning\wav2vec\_retraining\src\data\hdf5\_to\_validation.py”, line 97, in   
> dict\_valid = datasets.Dataset.from\_dict({‘input\_values’: v\_array\_values})  
> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\site-packages\datasets\arrow\_dataset.py”, line 859, in from\_dict  
> pa\_table = InMemoryTable.from\_pydict(mapping=mapping)  
> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\site-packages\datasets\table.py”, line 750, in from\_pydict  
> return cls(pa.Table.from\_pydict(\*args, \*\*kwargs))  
> File “pyarrow\table.pxi”, line 3625, in pyarrow.lib.Table.from\_pydict  
> File “pyarrow\table.pxi”, line 5150, in pyarrow.lib.\_from\_pydict  
> File “pyarrow\array.pxi”, line 342, in pyarrow.lib.asarray  
> File “pyarrow\array.pxi”, line 230, in pyarrow.lib.array  
> File “pyarrow\array.pxi”, line 110, in pyarrow.lib.\_handle\_arrow\_array\_protocol  
> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\site-packages\datasets\arrow\_writer.py”, line 180, in **arrow\_array**  
> out = list\_of\_np\_array\_to\_pyarrow\_listarray(data)  
> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\site-packages\datasets\features\features.py”, line 1330, in list\_of\_np\_array\_to\_pyarrow\_listarray  
> return list\_of\_pa\_arrays\_to\_pyarrow\_listarray(  
> File “C:\Users\<username\>\miniforge3\envs\wav2vec\_pretraining\lib\site-packages\datasets\features\features.py”, line 1322, in list\_of\_pa\_arrays\_to\_pyarrow\_listarray  
> offsets = pa.array(offsets, type=pa.int32())  
> File “pyarrow\array.pxi”, line 312, in pyarrow.lib.array  
> File “pyarrow\array.pxi”, line 83, in pyarrow.lib.\_ndarray\_to\_array  
> OverflowError: Python int too large to convert to C long

Do you have an idea about what could cause this error? I am on Windows, but I have an analog error on a Linux device too.  
Thanks!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [March 15, 2023, 5:45pm UTC](https://discuss.huggingface.co/t/pyarrow-lib-floatarray-did-not-recognize-python-value-type-when-inferring-an-arrow-data-type/33417/3 "2023-03-15T17:45:38Z")

</div>

Hi! Yes, `from_dict` does not currently support PyArrow arrays as column values, but it should, so I’ll fix this for the next release.

To fix the overflow error, we need to merge [support LargeListArray in pyarrow by xwwwwww · Pull Request #4800 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/4800), which adds support for the large lists. However, before merging it, we need to come up with a cleaner API for large lists. I hope to find some time to address this before Datasets 3.0.

---

<div class="post-metadata">

**Author:** ![MarLac](https://avatars.discourse-cdn.com/v4/letter/m/bbce88/32.png) [@MarLac](https://discuss.huggingface.co/u/MarLac)\
**Post date:** [March 17, 2023, 9:49am UTC](https://discuss.huggingface.co/t/pyarrow-lib-floatarray-did-not-recognize-python-value-type-when-inferring-an-arrow-data-type/33417/4 "2023-03-17T09:49:21Z")

</div>

Thank you for your response!  
I removed the conversion into pyarrow array and the values which do not reach the maximum length (to have all my data with the same size/shape).  
That works with these changes 🙂