# ArrowBasedBuilder versus GeneratorDBasedBuilder

**URL:** <https://discuss.huggingface.co/t/arrowbasedbuilder-versus-generatordbasedbuilder/29423>\
**Category:** 🤗Datasets\
**Created:** [January 12, 2023, 7:04am UTC](https://discuss.huggingface.co/t/arrowbasedbuilder-versus-generatordbasedbuilder/29423 "2023-01-12T07:04:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sl02](https://avatars.discourse-cdn.com/v4/letter/s/ba9def/32.png) [@sl02](https://discuss.huggingface.co/u/sl02)\
**Post date:** [January 12, 2023, 7:04am UTC](https://discuss.huggingface.co/t/arrowbasedbuilder-versus-generatordbasedbuilder/29423/1 "2023-01-12T07:04:57Z")

</div>

Could you please enumerate pros and cons for both these dataset builder classes. I couldn’t find anything in the documentation. When would I prefer one over the other. Is ArrowBasedBuilder more performant for large datasets?

Thank you!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 13, 2023, 4:08pm UTC](https://discuss.huggingface.co/t/arrowbasedbuilder-versus-generatordbasedbuilder/29423/2 "2023-01-13T16:08:26Z")

</div>

Yes `ArrowBasedBuilder` is more performant in general because datasets are saved in Arrow format.

Generating a dataset file from a `GeneratorBasedBuilder` requires to convert the data to Arrow when writing on disk.

Therefore it’s a good idea to use the `ArrowBasedBuilder` whenever you have a big dataset and you’re able to load your data in Arrow tables using `pyarrow`

---

<div class="post-metadata">

**Author:** ![Ryoo72](https://avatars.discourse-cdn.com/v4/letter/r/3da27b/32.png) [@Ryoo72](https://discuss.huggingface.co/u/Ryoo72)\
**Post date:** [February 7, 2025, 9:25am UTC](https://discuss.huggingface.co/t/arrowbasedbuilder-versus-generatordbasedbuilder/29423/3 "2025-02-07T09:25:57Z")

</div>

@lhoestq  
I want to create a large dataset (\>4M) that includes large-size images such as infographics. Could you advise on the best use case? It seems py-arrow is struggling to handle large amounts of PIL images, and uploading to hub is also challenging. Currently, I’m thinking of just inserting image paths as strings and loading them in the collate function - is this the best approach? Alternatively, I’m also considering using GeneratorBasedBuilder.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 7, 2025, 1:49pm UTC](https://discuss.huggingface.co/t/arrowbasedbuilder-versus-generatordbasedbuilder/29423/4 "2025-02-07T13:49:06Z")

</div>

Hi ! is it struggling because it takes a lot of memory ?

You can already store image paths and use the Image type (it decodes images on the fly):

```python
ds = Dataset.from_dict({"image": ["path/to/img0.png", ...]})
ds = ds.cast_column("image", Image())

```

and if you push\_to\_hub it will only load maximum 500MB in memory at a time to upload parquet files one by one.

---

<div class="post-metadata">

**Author:** ![Ryoo72](https://avatars.discourse-cdn.com/v4/letter/r/3da27b/32.png) [@Ryoo72](https://discuss.huggingface.co/u/Ryoo72)\
**Post date:** [February 8, 2025, 7:31pm UTC](https://discuss.huggingface.co/t/arrowbasedbuilder-versus-generatordbasedbuilder/29423/5 "2025-02-08T19:31:41Z")

</div>

@lhoestq

First of all, I want to express my sincere gratitude for your dedication to the community. I have three questions I’d like to ask:

1. I’ve created an Arrow-format dataset with **a list of images** as features at [here](https://huggingface.co/datasets/Ryoo72/DocStruct4M), using the [DocStruct4M dataset](https://huggingface.co/datasets/mPLUG/DocStruct4M/tree/main) which is originally about 400GB. However, despite only changing the annotation format, the size has increased to over 800GB. Since the uncompressed images themselves take up less than 400GB, I don’t believe this is a compression issue. I’m curious about why this size increase occurs. I noticed that a similar implementation at [here](https://huggingface.co/datasets/jp1924/DocStruct4M) only takes up around 300GB.

2. I’m trying to process “a list of images” in an Arrow file. The `ds.cast_column("image", Image())` method doesn’t seem to work in this case. Could you advise on how to handle this?  
I’m wondering if my current approach is correct: `my_dataset = dataset_dict['validation'].cast_column("images",[datasets.Image()])`

3. Looking at [datasets.Image](https://github.com/huggingface/datasets/blob/3.2.0/src/datasets/features/image.py), I notice that when `encode=False`, it still reads the file path using PIL. I’m wondering what advantages datasets.Image offers compared to simply storing file paths as strings.

Thank you again for your time and assistance.