# Map() function freezes on large dataset **URL:** \ **Category:** 🤗Datasets\ **Created:** [March 7, 2023, 9:27am UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224 "2023-03-07T09:27:49Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![BelalElhossany](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/belalelhossany/32/14551_2.png) [@BelalElhossany](https://discuss.huggingface.co/u/BelalElhossany)\ **Post date:** [March 7, 2023, 9:27am UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224/1 "2023-03-07T09:27:49Z")
Hi, I’m trying to use map on a dataset of size about 100GB, it hangs every time. I tried a lot of parameters combinations but it always hangs. I cannot even use for loop, values of the dictionary are not modified in a loop. But, the for loop doesn’t hang it only has no effect. I think the problem is in the I/O operations done in the map function, but I don’t know what the problem exactly is. This is the last error after hanging for 15 mins at 10% ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/9bed6edc7b137f77e33f9ec26dece3b835366c01.png) ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\ **Post date:** [March 15, 2023, 1:42pm UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224/2 "2023-03-15T13:42:12Z")
Hi! This error seems to come from [`ipython`](https://stackoverflow.com/a/74846739) bug, not from `datasets`. It would help if you could verify this by running the code outside the `ipython` env. ---
**Author:** ![BelalElhossany](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/belalelhossany/32/14551_2.png) [@BelalElhossany](https://discuss.huggingface.co/u/BelalElhossany)\ **Post date:** [March 15, 2023, 2:01pm UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224/3 "2023-03-15T14:01:03Z")
@mariosasko I did, it ran to completion but I think there is a huge overhead. I can iterate on the dataset in 10 mins, and it takes 30 mins to save the dataset on the disk. but the map function took about 4 hrs to finish. I don’t know what the problem is exactly but I think it’s a disk bottleneck as map tries to read data from the disk and save to the cache simultaneously. ---
**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\ **Post date:** [March 15, 2023, 2:09pm UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224/4 "2023-03-15T14:09:20Z")
You can probably make the `map` faster by reducing the number of processes to, e.g., `num_proc=os.cpu_count()`. ---
**Author:** ![BelalElhossany](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/belalelhossany/32/14551_2.png) [@BelalElhossany](https://discuss.huggingface.co/u/BelalElhossany)\ **Post date:** [March 15, 2023, 2:17pm UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224/5 "2023-03-15T14:17:07Z")
@mariosasko Great, thanks! ---
**Author:** ![whoismikha](https://avatars.discourse-cdn.com/v4/letter/w/48db29/32.png) [@whoismikha](https://discuss.huggingface.co/u/whoismikha)\ **Post date:** [September 8, 2023, 6:28pm UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224/6 "2023-09-08T18:28:16Z")
Is there a way to find out more precisely what the bottleneck is? My num\_proc is a t cpu\_count but it still takes unreasonably long to save at the end of processing (or when `writer_batch_size` is reached. Is it possible to disable the saving to disk entirely and only keep the map result in memory? `keep_in_memory` sounds like it does that but doesn’t seem to actually do it. ---
**Author:** ![whoismikha](https://avatars.discourse-cdn.com/v4/letter/w/48db29/32.png) [@whoismikha](https://discuss.huggingface.co/u/whoismikha)\ **Post date:** [September 8, 2023, 6:31pm UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224/7 "2023-09-08T18:31:25Z")
for me it for some reason only happens when I create PIL image columns during the mapping fn. If instead I convert PIL images to bytes, the map doesn’t freeze. Same exact dataset. @lhoestq do you have ideas? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [September 10, 2023, 11:13am UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224/8 "2023-09-10T11:13:28Z")
Could you share the code of the function you pass to `map` ? And also the version where you convert PIL Images to bytes ? Can you also share some information about your operating system (windows/max/linux ?) and your version of `pyarrow` and `datasets` ? ---
**Author:** ![ReatKay](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/reatkay/32/18831_2.png) [@ReatKay](https://discuss.huggingface.co/u/ReatKay)\ **Post date:** [September 10, 2023, 4:49pm UTC](https://discuss.huggingface.co/t/map-function-freezes-on-large-dataset/33224/9 "2023-09-10T16:49:06Z")
```auto # Disable disk caching from datasets import set_caching_enabled set_caching_enabled(False) ``` Regarding the num\_procs: What environment (OS)? And what functionality are u actually using in your map call(s)? Iterating a dataset is one thing, thats simply reading over a dataset, basically: Exactly what a dataset is made for, during mapping you actually do alterations. Simplest solution: After you did all your mapping, then save your dataset fully processed and usable to disk - and load it already mapped and usable from disk when you need it. Finally NVMe can really shine -laugh-