# Progress bar of dataset.map with num\_proc\>1 hangs

**URL:** <https://discuss.huggingface.co/t/progress-bar-of-dataset-map-with-num-proc-1-hangs/64776>\
**Category:** 🤗Datasets\
**Created:** [December 6, 2023, 10:58am UTC](https://discuss.huggingface.co/t/progress-bar-of-dataset-map-with-num-proc-1-hangs/64776 "2023-12-06T10:58:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![wjassim](https://avatars.discourse-cdn.com/v4/letter/w/7feea3/32.png) [@wjassim](https://discuss.huggingface.co/u/wjassim)\
**Post date:** [December 6, 2023, 10:58am UTC](https://discuss.huggingface.co/t/progress-bar-of-dataset-map-with-num-proc-1-hangs/64776/1 "2023-12-06T10:58:28Z")

</div>

Hello,

I’m currently experiencing an issue with my code. Here’s the code snippet:

```
  df = pd.read_csv(self.args["csv_input"])
  ds = Dataset.from_pandas(df)

  ds = ds.map(
    self.process_data,
    num_proc=self.args["num_workers"],
    with_indices=True,
    batched=True,
    batch_size=int(len(df) / self.args["num_workers"]),
    load_from_cache_file=True,
    desc="Processing Data.....",
)

```

However, the progress bar hangs, and the status is not updating:

```
Processing Data... (num_proc=10): 0%| | 0/520 [00:00<?, ? examples/s]

```

I’m using Python 3.10.13 and datasets-2.15.1.dev0. Any suggestions on resolving this issue would be greatly appreciated.

Thanks a million!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [December 6, 2023, 4:36pm UTC](https://discuss.huggingface.co/t/progress-bar-of-dataset-map-with-num-proc-1-hangs/64776/2 "2023-12-06T16:36:17Z")

</div>

It is hard to tell what the problem is without the complete code. If it works in the single-process mode, then this most likely means you don’t have enough RAM for `num_proc=10` (try to reduce `writer_batch_size` and `batch_size` in the `map` if that’s the case).

---

<div class="post-metadata">

**Author:** ![wjassim](https://avatars.discourse-cdn.com/v4/letter/w/7feea3/32.png) [@wjassim](https://discuss.huggingface.co/u/wjassim)\
**Post date:** [December 6, 2023, 8:07pm UTC](https://discuss.huggingface.co/t/progress-bar-of-dataset-map-with-num-proc-1-hangs/64776/3 "2023-12-06T20:07:33Z")

</div>

Thank you for your reply.

I tried running the code with a single core ( num\_proc=1) but still have the same issue.

I think I have enough RAM but I will try with reducing the writer\_batch\_size and batch\_size parameters to see how it goes.

Thank you again