# Working with large datasets - cache issues

**URL:** <https://discuss.huggingface.co/t/working-with-large-datasets-cache-issues/11687>\
**Category:** 🤗Datasets\
**Created:** [November 14, 2021, 1:09pm UTC](https://discuss.huggingface.co/t/working-with-large-datasets-cache-issues/11687 "2021-11-14T13:09:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![vitvit](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vitvit](https://discuss.huggingface.co/u/vitvit)\
**Post date:** [November 14, 2021, 1:09pm UTC](https://discuss.huggingface.co/t/working-with-large-datasets-cache-issues/11687/1 "2021-11-14T13:09:36Z")

</div>

The .map() function creates a cache file 100 times larger then the original dataset file.  
Can this behaviour be somehow avoided?  
Note that it happens when using “num\_proc=48”

---

<div class="post-metadata">

**Author:** ![woqucc](https://avatars.discourse-cdn.com/v4/letter/w/e480ec/32.png) [@woqucc](https://discuss.huggingface.co/u/woqucc)\
**Post date:** [June 1, 2022, 2:54am UTC](https://discuss.huggingface.co/t/working-with-large-datasets-cache-issues/11687/2 "2022-06-01T02:54:25Z")

</div>

I also encountered this today with datasets=2.2.1 and I’m looking for a solution for that. I processed a 3.7G JSON file by the map function with ‘num\_proc=32’, and more than 300G arrow files were created in the ‘/tmp’ dictionary. Of course, I got a No space on the device error.