# LoadDataSet pyarrow.lib.ArrowCapacityError

**URL:** <https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328>\
**Category:** 🤗Datasets\
**Created:** [January 12, 2025, 4:14am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328 "2025-01-12T04:14:59Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dammond](https://avatars.discourse-cdn.com/v4/letter/d/c37758/32.png) [@Dammond](https://discuss.huggingface.co/u/Dammond)\
**Post date:** [January 12, 2025, 4:14am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328/1 "2025-01-12T04:14:59Z")

</div>

I use

> data\_set = load\_dataset(self.data\_file\_path, cache\_dir=cache\_dir, split=“train”)

Report when loading dataset (approximately 84GB)

> pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 10761561509，

Try setting up according to the help provided in other posts

> set(data\_set[“hash”])

I still haven’t solved the above problem. Do you have any ways to help me solve it? Thank you!

My version information is as follows

- `datasets` version: 3.2.0
- Platform: Linux-4.19.91-014.15-kangaroo.alios7.x86\_64-x86\_64-with-glibc2.35
- Python version: 3.11.10
- `huggingface_hub` version: 0.26.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.2.0

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 12, 2025, 4:51am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328/2 "2025-01-12T04:51:03Z")

</div>

Apparently, this is an issue with PyArrow, and although some of it has been resolved, it still seems to be unresolved. @lhoestq

> <https://github.com/huggingface/datasets/issues/5695>
>
> \### Describe the bug
> 
> Calling \`datasets.load\_dataset\` to load the (publicly avai…lable) dataset \`theodor1289/wit\` fails with \`pyarrow.lib.ArrowNotImplementedError\`.
> 
> \### Steps to reproduce the bug
> 
> Steps to reproduce this behavior:
> 
> 1. \`!pip install datasets\`
> 2. \`!huggingface-cli login\`
> 3. This step will throw the error (it might take a while as the dataset has ~170GB):
> \`\`\`python
> from datasets import load\_dataset
> dataset = load\_dataset("theodor1289/wit", "train", use\_auth\_token=True)
> \`\`\`
> 
> Stack trace:
> \`\`\`
> (torch-multimodal) bash-4.2$ python test.py 
> Downloading and preparing dataset None/None to /cluster/work/cotterell/tamariucai/HuggingfaceDatasets/theodor1289\_\_\_parquet/theodor1289--wit-7a3e984414a86a0f/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
> Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 \[00:00\<00:00, 491.68it/s\]
> Extracting data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 \[00:00\<00:00, 16.93it/s\]
> Traceback (most recent call last):                                                                                                                                                                                                    
> File "/cluster/home/tamariucai/.local/lib/python3.10/site-packages/datasets/builder.py", line 1860, in \_prepare\_split\_single
> for \_, table in generator:
> File "/cluster/home/tamariucai/.local/lib/python3.10/site-packages/datasets/packaged\_modules/parquet/parquet.py", line 69, in \_generate\_tables
> for batch\_idx, record\_batch in enumerate(
> File "pyarrow/\_parquet.pyx", line 1323, in iter\_batches
> File "pyarrow/error.pxi", line 121, in pyarrow.lib.check\_status
> pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
> File "/cluster/work/cotterell/tamariucai/multimodal-mirror/examples/test.py", line 2, in \<module\>
> dataset = load\_dataset("theodor1289/wit", "train", use\_auth\_token=True)
> File "/cluster/home/tamariucai/.local/lib/python3.10/site-packages/datasets/load.py", line 1791, in load\_dataset
> builder\_instance.download\_and\_prepare(
> File "/cluster/home/tamariucai/.local/lib/python3.10/site-packages/datasets/builder.py", line 891, in download\_and\_prepare
> self.\_download\_and\_prepare(
> File "/cluster/home/tamariucai/.local/lib/python3.10/site-packages/datasets/builder.py", line 986, in \_download\_and\_prepare
> self.\_prepare\_split(split\_generator, \*\*prepare\_split\_kwargs)
> File "/cluster/home/tamariucai/.local/lib/python3.10/site-packages/datasets/builder.py", line 1748, in \_prepare\_split
> for job\_id, done, content in self.\_prepare\_split\_single(
> File "/cluster/home/tamariucai/.local/lib/python3.10/site-packages/datasets/builder.py", line 1893, in \_prepare\_split\_single
> raise DatasetGenerationError("An error occurred while generating the dataset") from e
> datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
> 
> \`\`\`
> 
> \### Expected behavior
> 
> The dataset is loaded in variable \`dataset\`.
> 
> \### Environment info
> 
> \- \`datasets\` version: 2.11.0
> \- Platform: Linux-3.10.0-1160.80.1.el7.x86\_64-x86\_64-with-glibc2.17
> \- Python version: 3.10.4
> \- Huggingface\_hub version: 0.13.3
> \- PyArrow version: 11.0.0
> \- Pandas version: 1.5.3

---

<div class="post-metadata">

**Author:** ![Dammond](https://avatars.discourse-cdn.com/v4/letter/d/c37758/32.png) [@Dammond](https://discuss.huggingface.co/u/Dammond)\
**Post date:** [January 12, 2025, 4:59am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328/3 "2025-01-12T04:59:33Z")

</div>

Yes, I have seen similar posts with the same issue:

> [Minhash Deduplication - #11 by conceptofmind](https://discuss.huggingface.co/t/minhash-deduplication/19992/11)

But I tried this method and it didn’t solve the error.  
May I ask if there is any way you can help me solve this problem?  
Thank you!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 12, 2025, 5:18am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328/4 "2025-01-12T05:18:00Z")

</div>

How about trying .shard()?

> [@How to load a large hf dataset efficiently?](https://discuss.huggingface.co/t/how-to-load-a-large-hf-dataset-efficiently/69288/3):
>
> Thank you for the reply, following your suggestion is there any way in hf dataset, split into multiple shards to be loaded in parallel?

---

<div class="post-metadata">

**Author:** ![Dammond](https://avatars.discourse-cdn.com/v4/letter/d/c37758/32.png) [@Dammond](https://discuss.huggingface.co/u/Dammond)\
**Post date:** [January 12, 2025, 5:45am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328/5 "2025-01-12T05:45:20Z")

</div>

Thank you for your reply.  
Is. shard() partitioned only when generating the load\_dataset() object

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/9/7/974bad8338db8548f4684f065d29c44640b5f336.png)  
But this error occurred during load\_dataset()  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/c/7/c7ada341cf480ef7d0b968f53fe220dd2210362b.png)  
 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/5/3/53306c65d81820ea4936096b3c7642a1e8554411.png)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [January 12, 2025, 5:58am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328/6 "2025-01-12T05:58:49Z")

</div>

It may also be another limitation of PyArrow. If you set num\_shards to around 20, maybe it will work… I hope it does.

> <https://github.com/ray-project/ray/issues/48419>
>
> \### Description
> 
> I have 1100 shards of 3 Gbytes of data each that I read with …read\_text. I cannot find any way to process them with ray.data.
> 
> I consistently get these errors:
> 
> \`\`\`
> pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
> \`\`\`
> 
> The problem seems to be that pyarrow internally is limited to 2 Gbyte offsets. This seems like a pretty serious limitation given modern processors and datasets. It wouldn't be quite so serious if the ray.data.read\_... routines consistently had a chunk\_size or block\_size argument that would permit on the fly splitting of files into multiple blocks, but they don't.
> 
> I don't even seem to be able to handle this by using \`read\_binary\_file\` with \`flat\_map\`, since even the output of \`read\_binary\_file\` is represented as a PyArrow table and can't hold the raw binary data.
> 
> \`\`\`
> File "pyarrow/array.pxi", line 358, in pyarrow.lib.array
> File "pyarrow/array.pxi", line 85, in pyarrow.lib.\_ndarray\_to\_array
> File "pyarrow/error.pxi", line 92, in pyarrow.lib.check\_status
> pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2626945332
> \`\`\`
> 
> My suggestions would be:
> 
> (1) add optional block\_size arguments to all readers
> (2) consider the support of a different/additional internal format other than PyArrow, a format not limited by 32 bit offsets
> 
> Somewhat related:
> 
> (3) instead of dealing with data partitioning in terms of total number of blocks (which is hard to do for datasets of unknown size), handle data partitioning in terms of block sizes.
> (4) implement a streaming\_repartition(block\_size)
> 
> \### Use case
> 
> I'm trying to read a sharded JSONL datasets whose shards happen to expand into data larger than 2GB when decompressed. Processing this dataset with ray.data seems impossible, since it is impossible even to read it with read\_text.

---

<div class="post-metadata">

**Author:** ![Dammond](https://avatars.discourse-cdn.com/v4/letter/d/c37758/32.png) [@Dammond](https://discuss.huggingface.co/u/Dammond)\
**Post date:** [January 12, 2025, 6:33am UTC](https://discuss.huggingface.co/t/loaddataset-pyarrow-lib-arrowcapacityerror/135328/7 "2025-01-12T06:33:25Z")

</div>

I have already set num\_stards to 100, but the same error still exists

> data\_set = load\_dataset(self.data\_file\_path, cache\_dir=cache\_dir, split=“train”)  
> data\_set = data\_set.shard(num\_shards=100, index=0)

It seems that this error already exists when executing load\_dataset()

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/3/f/3f4b7a3d153198328c82de8c631f53a57068b34e.png)