# Arrowmemoryerror: realloc of size 32 GB failed

**URL:** <https://discuss.huggingface.co/t/arrowmemoryerror-realloc-of-size-32-gb-failed/28488>\
**Category:** 🤗Datasets\
**Created:** [December 27, 2022, 3:31am UTC](https://discuss.huggingface.co/t/arrowmemoryerror-realloc-of-size-32-gb-failed/28488 "2022-12-27T03:31:38Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chiintan](https://avatars.discourse-cdn.com/v4/letter/c/a8b319/32.png) [@Chiintan](https://discuss.huggingface.co/u/Chiintan)\
**Post date:** [December 27, 2022, 3:31am UTC](https://discuss.huggingface.co/t/arrowmemoryerror-realloc-of-size-32-gb-failed/28488/1 "2022-12-27T03:31:38Z")

</div>

I am trying to encode an image dataset of size 20000 images. But when the encoding reaches at 69% irrespective of the batch size it generates an error displaying realloc of size 32 gb failed. I even tried to increase the batch size but the error still persists. Pyarrow version that is installed is 10.0.1, datasets version is 2.7.1.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [January 3, 2023, 10:48am UTC](https://discuss.huggingface.co/t/arrowmemoryerror-realloc-of-size-32-gb-failed/28488/2 "2023-01-03T10:48:55Z")

</div>

Hi ! How do you create your dataset ?

Right now every dataset is loaded from disk using memory mapping to not fill your RAM. However datasets created from in-memory data currently stay in memory.

So if you used `Dataset.from_dict` for example you may want to write your dataset to disk to avoid filling up your RAM. You can use `ds.save_to_disk()` and reload it with `load_form_disk()` before calling your map function

---

<div class="post-metadata">

**Author:** ![Chiintan](https://avatars.discourse-cdn.com/v4/letter/c/a8b319/32.png) [@Chiintan](https://discuss.huggingface.co/u/Chiintan)\
**Post date:** [January 6, 2023, 10:18am UTC](https://discuss.huggingface.co/t/arrowmemoryerror-realloc-of-size-32-gb-failed/28488/3 "2023-01-06T10:18:28Z")

</div>

Thank you @lhoestq for the Solution. Will definitely try it if it works.