# Num\_proc is not working with map

**URL:** <https://discuss.huggingface.co/t/num-proc-is-not-working-with-map/45641>\
**Category:** Beginners\
**Created:** [July 4, 2023, 6:11pm UTC](https://discuss.huggingface.co/t/num-proc-is-not-working-with-map/45641 "2023-07-04T18:11:22Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![mohanvrk](https://avatars.discourse-cdn.com/v4/letter/m/258eb7/32.png) [@mohanvrk](https://discuss.huggingface.co/u/mohanvrk)\
**Post date:** [July 4, 2023, 6:11pm UTC](https://discuss.huggingface.co/t/num-proc-is-not-working-with-map/45641/1 "2023-07-04T18:11:22Z")

</div>

Hi All,

I have been struggling to make the map tokenization parallel, however, I couldn’t make it.

I request, could you please suggest me in this regard.

Here is the example code.

training\_dataset = dataset.map(  
lambda example, idx: tokenize(  
example,  
idx,  
vocab,  
in\_df.columns,  
decoder\_dataset,  
in\_out\_idx,  
output\_max\_length,  
),  
remove\_columns=dataset.column\_names,  
with\_indices=True,  
num\_proc = 40)

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [July 6, 2023, 12:51pm UTC](https://discuss.huggingface.co/t/num-proc-is-not-working-with-map/45641/2 "2023-07-06T12:51:30Z")

</div>

`num_proc` only makes sense for slow tokenizers. If `tokenizer.is_fast` returns `True`, you should use `map` in the batched mode (fast tokenizers automatically tokenize a batch of samples in parallel) and set `num_proc=None` (fast tokenizers are written in Rust, and their multiprocessing module does not support Python multiprocessing) to parallelize the processing.

---

<div class="post-metadata">

**Author:** ![mohanvrk](https://avatars.discourse-cdn.com/v4/letter/m/258eb7/32.png) [@mohanvrk](https://discuss.huggingface.co/u/mohanvrk)\
**Post date:** [July 6, 2023, 1:18pm UTC](https://discuss.huggingface.co/t/num-proc-is-not-working-with-map/45641/3 "2023-07-06T13:18:33Z")

</div>

Hi @mariosasko,

Thanks for the reply.

However, this current tokenizer is a custom one written purely in python. Is there any possibility to parallelize the mapping process in this case.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [July 6, 2023, 1:54pm UTC](https://discuss.huggingface.co/t/num-proc-is-not-working-with-map/45641/4 "2023-07-06T13:54:37Z")

</div>

Yes, `num_proc` (but not too high, e.g., `os.cpu_count()` is a good number), and `batched=True` should yield the best performance in that scenario.

---

<div class="post-metadata">

**Author:** ![mohanvrk](https://avatars.discourse-cdn.com/v4/letter/m/258eb7/32.png) [@mohanvrk](https://discuss.huggingface.co/u/mohanvrk)\
**Post date:** [July 6, 2023, 3:57pm UTC](https://discuss.huggingface.co/t/num-proc-is-not-working-with-map/45641/5 "2023-07-06T15:57:55Z")

</div>

Thanks @mariosasko,  
Let me give a try and get back to you.

---

<div class="post-metadata">

**Author:** ![wrexial33](https://avatars.discourse-cdn.com/v4/letter/w/f08c70/32.png) [@wrexial33](https://discuss.huggingface.co/u/wrexial33)\
**Post date:** [April 15, 2024, 4:13pm UTC](https://discuss.huggingface.co/t/num-proc-is-not-working-with-map/45641/6 "2024-04-15T16:13:42Z")

</div>

If you try that and it doesn’t work, try passing the custom tokenizer in as a fn\_kwargs. The .map function does not play nice with globally defined variables. Ex.

```auto
my_dataset = my_dataset.map(
    lambda example, idx: custom_tokenizer(example, idx, 
              input_vocab, 
              input_df.columns,
              decoder, 
              in_out_index, 
              max_length,), 
    remove_columns=dataset.column_names, 
    with_indicies=True, 
    num_proc=40, 
    fn_kwargs={
              'custom_tokenizer': tokenzier, 
              'input_vocab': vocab, 
              'input_df': in_df, 
              'decoder': decoder_dataset,
              'in_out_index': in_out_idx, 
              'max_length': output_max_length})

```

I would personally use a def function instead and feed in an array containing the parameters.

---

<div class="post-metadata">

**Author:** ![G999n](https://avatars.discourse-cdn.com/v4/letter/g/5f8ce5/32.png) [@G999n](https://discuss.huggingface.co/u/G999n)\
**Post date:** [August 23, 2025, 3:15pm UTC](https://discuss.huggingface.co/t/num-proc-is-not-working-with-map/45641/7 "2025-08-23T15:15:27Z")

</div>

Thanks a lot bud! I was stuck in it for days. I reset password and logged in just to thank you. Haha