# Dataset map function takes forever to run!

**URL:** <https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694>\
**Category:** 🤗Datasets\
**Created:** [April 5, 2023, 9:39pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694 "2023-04-05T21:39:52Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![harish3110](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/harish3110/32/15205_2.png) [@harish3110](https://discuss.huggingface.co/u/harish3110)\
**Post date:** [April 5, 2023, 9:39pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/1 "2023-04-05T21:39:52Z")

</div>

I’m trying to pre-process my dataset for the Donut model and despite completeing the mapping it is running for about 100 mins -.-. I ran this with `num_proc=2`, not sure if setting it to all cpu cores would make much of a difference.

Any idea of how to fix this?

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/da72f1c12f865e69ee21050740cc147cf6feba09.png)

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [April 6, 2023, 1:57pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/2 "2023-04-06T13:57:04Z")

</div>

Hi! What does `processor.tokenizer.is_fast` return? If the returned value is `True`, it’s better not to use the `num_proc` parameter in `map` to benefit from the tokenizer’s parallelism. The “fast” tokenizers are written in Rust and process data in parallel by default, but this does work well in multi-process Python code, so we disable the “fast” tokenizers’ parallelism when `num_proc>1` to avoid deadlocks.

Also, setting the `return_tensors` parameter to `np` should make the transform faster as PyArrow natively supports NumPy 1-D arrays, which avoids the `torch` → `np` conversion step.

---

<div class="post-metadata">

**Author:** ![harish3110](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/harish3110/32/15205_2.png) [@harish3110](https://discuss.huggingface.co/u/harish3110)\
**Post date:** [April 6, 2023, 2:29pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/3 "2023-04-06T14:29:00Z")

</div>

Thanks @mariosasko. Yes, the tokenizer is fast. The reason I ran this `num_proc>1` because without it the code ate up all my RAM(32gb) and the kernel kept dying.

---

<div class="post-metadata">

**Author:** ![Pensive](https://avatars.discourse-cdn.com/v4/letter/p/f07891/32.png) [@Pensive](https://discuss.huggingface.co/u/Pensive)\
**Post date:** [October 19, 2023, 3:22am UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/4 "2023-10-19T03:22:33Z")

</div>

Was this problem ever solved? I played around with different settings mentioned in this thread and it changes nothing.

dataset.map stalls out around 80% of the dataset completion and then gets steadily slower and slower until it pretty much stops at 97% complete.

I tried to cut a smaller piece of the dataset to try, but this behavior persists on a smaller set too.

At the end it seems only one process is running and the rest are idle.

After 27 hours I just ctl-C out of it.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [October 19, 2023, 1:57pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/5 "2023-10-19T13:57:58Z")

</div>

@Pensive Even with `num_proc=None` (or `num_proc=1`)? If that’s the case, can you interrupt the `map` (with `num_proc=1`) and paste the printed error stack trace?

---

<div class="post-metadata">

**Author:** ![Pensive](https://avatars.discourse-cdn.com/v4/letter/p/f07891/32.png) [@Pensive](https://discuss.huggingface.co/u/Pensive)\
**Post date:** [October 19, 2023, 3:04pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/6 "2023-10-19T15:04:13Z")

</div>

With None or 1 it is slow from the get go. If I interrupt it, it is somewhere in the multiprocessing.contex.timeout.

It is not even consistent.

If I put num\_proc=30, a lot of time it stalls right away, and is slow like with None or 1

Sometimes, it starts pretty quickly runs to 80% slows down, and then is extremely slow and never finishes.

When I interrupted at that point, the trace had a line about being in the semaphore function.

---

<div class="post-metadata">

**Author:** ![Pensive](https://avatars.discourse-cdn.com/v4/letter/p/f07891/32.png) [@Pensive](https://discuss.huggingface.co/u/Pensive)\
**Post date:** [October 19, 2023, 3:05pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/7 "2023-10-19T15:05:55Z")

</div>

Traceback (most recent call last):  
File “preprocess.py”, line 81, in   
dataset = dataset.map(partial(tokenize\_fn, tokenizer), batched=False, num\_proc=30, remove\_columns=[“text”, “meta”])  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/arrow\_dataset.py”, line 592, in wrapper  
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/arrow\_dataset.py”, line 557, in wrapper  
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/arrow\_dataset.py”, line 3189, in map  
for rank, done, content in iflatmap\_unordered(  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/utils/py\_utils.py”, line 1394, in iflatmap\_unordered  
[async\_result.get(timeout=0.05) for async\_result in async\_results]  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/utils/py\_utils.py”, line 1394, in   
[async\_result.get(timeout=0.05) for async\_result in async\_results]  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/pool.py”, line 767, in get  
raise TimeoutError  
multiprocess.context.TimeoutError

---

<div class="post-metadata">

**Author:** ![Pensive](https://avatars.discourse-cdn.com/v4/letter/p/f07891/32.png) [@Pensive](https://discuss.huggingface.co/u/Pensive)\
**Post date:** [October 19, 2023, 3:14pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/8 "2023-10-19T15:14:13Z")

</div>

KeyboardInterrupt  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/process.py”, line 315, in \_bootstrap  
self.run()  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/process.py”, line 108, in run  
self.\_target(\*self.\_args, \*\*self.\_kwargs)  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/pool.py”, line 114, in worker  
task = get()  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/queues.py”, line 358, in get  
with self.\_rlock:  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/synchronize.py”, line 101, in **enter**  
return self.\_semlock. **enter** ()  
KeyboardInterrupt  
Map (num\_proc=16): 82%|███████████████████████████████████████████████████████████████████████████████████▍ | 7614/9305 [00:28\<00:06, 262.96 examples/s]  
Traceback (most recent call last):  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/utils/py\_utils.py”, line 1380, in iflatmap\_unordered  
yield queue.get(timeout=0.05)  
File “”, line 2, in get  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/managers.py”, line 835, in \_callmethod  
kind, result = conn.recv()  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/connection.py”, line 253, in recv  
buf = self.\_recv\_bytes()  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/connection.py”, line 417, in \_recv\_bytes  
buf = self.\_recv(4)  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/connection.py”, line 382, in \_recv  
chunk = read(handle, remaining)  
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):  
File “preprocess.py”, line 81, in   
dataset = dataset.map(partial(tokenize\_fn, tokenizer), batched=False, num\_proc=16, remove\_columns=[“text”, “meta”])  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/arrow\_dataset.py”, line 592, in wrapper  
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/arrow\_dataset.py”, line 557, in wrapper  
out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs)  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/arrow\_dataset.py”, line 3189, in map  
for rank, done, content in iflatmap\_unordered(  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/utils/py\_utils.py”, line 1394, in iflatmap\_unordered  
[async\_result.get(timeout=0.05) for async\_result in async\_results]  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/datasets/utils/py\_utils.py”, line 1394, in   
[async\_result.get(timeout=0.05) for async\_result in async\_results]  
File “/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/multiprocess/pool.py”, line 767, in get  
raise TimeoutError  
multiprocess.context.TimeoutError

---

<div class="post-metadata">

**Author:** ![AnanthZeke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ananthzeke/32/48678_2.png) [@AnanthZeke](https://discuss.huggingface.co/u/AnanthZeke)\
**Post date:** [January 26, 2024, 5:44pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/9 "2024-01-26T17:44:43Z")

</div>

Hey @mariosasko @Pensive , is that problem solved .

---

<div class="post-metadata">

**Author:** ![Pensive](https://avatars.discourse-cdn.com/v4/letter/p/f07891/32.png) [@Pensive](https://discuss.huggingface.co/u/Pensive)\
**Post date:** [January 28, 2024, 6:15pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/10 "2024-01-28T18:15:57Z")

</div>

I know what the problem is and how to overcome it, but NOT solved in the dataset.map.

---

<div class="post-metadata">

**Author:** ![pryhotsky](https://avatars.discourse-cdn.com/v4/letter/p/53a042/32.png) [@pryhotsky](https://discuss.huggingface.co/u/pryhotsky)\
**Post date:** [January 28, 2024, 10:31pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/11 "2024-01-28T22:31:50Z")

</div>

So, how can I overcome it?  
Just met the same problem, with the latest library version. I’ve tried everything I could find, but nothing has worked 😕

---

<div class="post-metadata">

**Author:** ![Pensive](https://avatars.discourse-cdn.com/v4/letter/p/f07891/32.png) [@Pensive](https://discuss.huggingface.co/u/Pensive)\
**Post date:** [January 29, 2024, 12:28am UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/12 "2024-01-29T00:28:41Z")

</div>

The problem lies in the dependence of dataset.map on multiprocessing process pool. The standard process pool uses “fork” to create new processes. Depending on timing  
when processes are created using “fork,” locks sometimes get copied in a locked state, thus causing deadlocks. You would have to use “forkserver” instead to prevent this from happening.

---

<div class="post-metadata">

**Author:** ![lucadiliello](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lucadiliello/32/26763_2.png) [@lucadiliello](https://discuss.huggingface.co/u/lucadiliello)\
**Post date:** [May 8, 2024, 12:49pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/13 "2024-05-08T12:49:22Z")

</div>

I experienced a similar slowdown while processing a very large dataset (LAION-2B). Script started processing very fast but eventually hanged at around 40%. I discovered the issue was related to having too many open files (limit on my instance was 512), while I had 256 parallel processes (probably with more than 2 files open per process). By scaling down processes to 128, script worked fine, while raising open file limits with ulimit didn’t work. I hope that this will be helpful.

---

<div class="post-metadata">

**Author:** ![andreasrath](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/andreasrath/32/29541_2.png) [@andreasrath](https://discuss.huggingface.co/u/andreasrath)\
**Post date:** [July 18, 2024, 10:57am UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/14 "2024-07-18T10:57:45Z")

</div>

Same here: we have an audio dataset with 9 rows. When call dataset.map(…) with 2 processes then the map function starts to execute but stalls immeadelty at the first code line inside the function to be executed - even if it is just a print statement.

We use latest transformers version `v4.42.3` and dataset version `2.20.0`

@mariosasko any ideas?

---

<div class="post-metadata">

**Author:** ![Hannan](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@Hannan](https://discuss.huggingface.co/u/Hannan)\
**Post date:** [July 21, 2024, 3:55pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/15 "2024-07-21T15:55:05Z")

</div>

I had the same problem when I do mapping for an audio dataset for `num_proc>1`. I traced the function and found it stucks when big audio array (big means even 1 second audio arrays) is just accessed (even just printing or assigning to a variable in function) but if another column of dataset that does not contain big audio array (like text of audio) is accessed or modified, it works in multiprocess mode properly.  
After days of struggling, accidentally I found by getting one of the samples of dataset by `next(iter(dataset))` just before the `dataset.map()`, it works properly on muliple processors.  
It is also recommended to run your python code with `torchrun yourcode.py`, while `python yourcode.py` would run the code but some `SIGTERM` logs of some processors are printed.

---

<div class="post-metadata">

**Author:** ![andreasrath](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/andreasrath/32/29541_2.png) [@andreasrath](https://discuss.huggingface.co/u/andreasrath)\
**Post date:** [July 21, 2024, 4:40pm UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/16 "2024-07-21T16:40:40Z")

</div>

Super, many thanks @Hannan !

---

<div class="post-metadata">

**Author:** ![Hannan](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@Hannan](https://discuss.huggingface.co/u/Hannan)\
**Post date:** [August 15, 2024, 11:37am UTC](https://discuss.huggingface.co/t/dataset-map-function-takes-forever-to-run/35694/17 "2024-08-15T11:37:36Z")

</div>

I found another issue in the usage of dataset.map() method on handling large audio datasets, regardless it is run by `python` or `accelerate`:  
The procedure will be freezed and takes forever (until a timeout happens) if the size of dataset is larger than a non-fixed amount.  
In my case, for a specific `num_proc=192`, dataset.map() works well if audio dataset has less than roughly 60k samples, but for datasets with more samples, it freezes somewhere after step 60k.  
I decreased `num_proc` to 128 and from then it freezes from around 45k.  
but as I mentioned, when I run dozens loops of dataset.map() but none of them exceeds that limit, everything works well.

When I run the code with `accelerate`, it just freezes but when I run it with `python` it raises the following error after a few minutes of freezing:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/8/0/802d668128c1de718a3c25b32b9cee33532f60a4.png)

It should be noted that I have checked the memory and space during the procedure and I’m sure there is no limits from them (it freezes when there is enough free memory and space)

@sanchit-gandhi while you have prepared many audio datasets for training whisper model , have you faced such issue?

@andreasrath