# Dataset.map hangs on tokenization (relatively small dataset)

**URL:** <https://discuss.huggingface.co/t/dataset-map-hangs-on-tokenization-relatively-small-dataset/16591>\
**Category:** 🤗Datasets\
**Created:** [April 7, 2022, 4:35pm UTC](https://discuss.huggingface.co/t/dataset-map-hangs-on-tokenization-relatively-small-dataset/16591 "2022-04-07T16:35:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lyma](https://avatars.discourse-cdn.com/v4/letter/l/e68b1a/32.png) [@Lyma](https://discuss.huggingface.co/u/Lyma)\
**Post date:** [April 7, 2022, 4:35pm UTC](https://discuss.huggingface.co/t/dataset-map-hangs-on-tokenization-relatively-small-dataset/16591/1 "2022-04-07T16:35:43Z")

</div>

Hello!

My Dataset is not huge at all: num\_rows: 198596 in the training set and 24825 in test and valid datasets each.  
I have 4 processes in Collab (output of !nproc).

Yet, when I’m running the dataset.map, at some point it hangs and never finishes running. Usually it hangs at the same %. Stopping it and re-running doesn’t help (yet, cached files are loaded properly)

I run dataset.map with the following arguments,  
tokenized\_ds = dataset.map(preprocess\_function, num\_proc=4, batched=True, remove\_columns=[‘name’])

The tokenizer used in the preprocess\_function is `AutoTokenizer.from_pretrained("distilbert-base-uncased")`, but I doubt it matters.

I would appreciate suggestions!

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [April 8, 2022, 4:46pm UTC](https://discuss.huggingface.co/t/dataset-map-hangs-on-tokenization-relatively-small-dataset/16591/2 "2022-04-08T16:46:07Z")

</div>

Hi! Does this only happen if `num_proc` is not 1 (or None)? Because if that’s not the case, then set `num_proc` to 1 for easier debugging. Also, feel free to share the (entire) traceback that you get after interrupting the process while waiting for it to finish.

---

<div class="post-metadata">

**Author:** ![Lyma](https://avatars.discourse-cdn.com/v4/letter/l/e68b1a/32.png) [@Lyma](https://discuss.huggingface.co/u/Lyma)\
**Post date:** [April 22, 2022, 9:35am UTC](https://discuss.huggingface.co/t/dataset-map-hangs-on-tokenization-relatively-small-dataset/16591/3 "2022-04-22T09:35:27Z")

</div>

Hi!

Thanks for the reply!

The problem was that my pieces of data were way too big in some rows (up to 15 MB (!)), so the batch didn’t fit in memory and the whole thing was crashing.

The problem was solved one step earlier, by splitting data to some chunks.