# Strange pyarrow error when extracting rows from a public dataset

**URL:** <https://discuss.huggingface.co/t/strange-pyarrow-error-when-extracting-rows-from-a-public-dataset/152889>\
**Category:** Intermediate\
**Created:** [April 29, 2025, 4:17pm UTC](https://discuss.huggingface.co/t/strange-pyarrow-error-when-extracting-rows-from-a-public-dataset/152889 "2025-04-29T16:17:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![OlivierSchmittSonar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olivierschmittsonar/32/46603_2.png) [@OlivierSchmittSonar](https://discuss.huggingface.co/u/OlivierSchmittSonar)\
**Post date:** [April 29, 2025, 4:17pm UTC](https://discuss.huggingface.co/t/strange-pyarrow-error-when-extracting-rows-from-a-public-dataset/152889/1 "2025-04-29T16:17:10Z")

</div>

Hi,

I’m trying to download and process rows from

> **[OpenCoder-LLM/opc-annealing-corpus at main](https://huggingface.co/datasets/OpenCoder-LLM/opc-annealing-corpus/tree/main/algorithmic_corpus)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

I’m using the generic load\_dataset function (using datasets = “==3.5.1”):  
`rows = load_dataset(path="OpenCoder-LLM/opc-annealing-corpus", name="algorithmic_corpus", split="train")`

I can download and generate the split:

```auto
Downloading data: 100%|██████████| 34/34 [00:00<00:00, 86.59files/s]
Generating train split: 10579390 examples [00:07, 1452745.44 examples/s]

```

But just after building the split it seems load\_dataset tries to load this file as an arrow file:

`https://huggingface.co/datasets/OpenCoder-LLM/opc-annealing-corpus/blob/main/algorithmic_corpus/state.json`

And this leads to an error:

```auto
ERROR: Failed to read file '/Users/<HOME>/.cache/huggingface/hub/datasets--OpenCoder-LLM--opc-annealing-corpus/snapshots/cb08b1b40bb19f88e3c4f48d6b4647bed588fc04/algorithmic_corpus/state.json' with error <class 'pyarrow.lib.ArrowInvalid'>: Not an Arrow file

pyarrow.lib.ArrowInvalid: Expected to read 538970747 metadata bytes, but only read 2131

```

Which makes sense because state.json is not an arrow file , it’s describing the dataset.  
I’d expect load\_dataset since it’s able to load the proper arrow files to not consider this Json file as an arrow file.  
Looks like a bug.

Any idea how to fix this?  
Thanks a lot.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 30, 2025, 1:42am UTC](https://discuss.huggingface.co/t/strange-pyarrow-error-when-extracting-rows-from-a-public-dataset/152889/2 "2025-04-30T01:42:22Z")

</div>

Hmm… PyArrow format datasets may not be loaded using the load\_dataset function.

> **[BAAI/Infinity-Instruct · 0625 Split Error: \`pyarrow.lib.ArrowInvalid:...](https://huggingface.co/datasets/BAAI/Infinity-Instruct/discussions/15)**
>
> Like the title says, we get the following error for the 0625 split: pyarrow.lib.ArrowInvalid: Expected to read 538970747 metadata bytes, but only read 1072 Which can be fixed temporarily using revi...

> hyxmmm  
> It seems that the arrow dataset is not supported to be loaded with load\_dataset(). We re-uploaded the dataset in parquet format. Please download it again. Sorry for the trouble.  
> [OpenCoder-LLM/opc-annealing-corpus · Met ArrowInvalid error](https://huggingface.co/datasets/OpenCoder-LLM/opc-annealing-corpus/discussions/4)  
> [`load_dataset` does not work with uploaded arrow file · Issue #3035 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/3035)

---

<div class="post-metadata">

**Author:** ![OlivierSchmittSonar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olivierschmittsonar/32/46603_2.png) [@OlivierSchmittSonar](https://discuss.huggingface.co/u/OlivierSchmittSonar)\
**Post date:** [April 30, 2025, 6:50am UTC](https://discuss.huggingface.co/t/strange-pyarrow-error-when-extracting-rows-from-a-public-dataset/152889/3 "2025-04-30T06:50:12Z")

</div>

Thanks!  
So now I have two options:

- use the parquet branch: [OpenCoder-LLM/opc-annealing-corpus at refs/convert/parquet](https://huggingface.co/datasets/OpenCoder-LLM/opc-annealing-corpus/tree/refs%2Fconvert%2Fparquet)
- try [Load](https://huggingface.co/docs/datasets/en/loading#arrow) and the data\_files options (requires playing with URLs and likely wildcards)

It’s a bit frustrating to see the repo contains all the metadata but a specific approach is required or the dataset needs to be duplicated in a another format.

A unified interface loading all sorts of dataset formats would be great; it seems almost implemented because the load\_dataset function loads all the arrow files by itself.

Might look into the code to see if I can comme with a change.  
Thanks again!  
Best.