# Cache & parallelize long tokenization step

**URL:** <https://discuss.huggingface.co/t/cache-parallelize-long-tokenization-step/25791>\
**Category:** 🤗Transformers\
**Created:** [November 9, 2022, 8:51am UTC](https://discuss.huggingface.co/t/cache-parallelize-long-tokenization-step/25791 "2022-11-09T08:51:21Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![AngledLuffa](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@AngledLuffa](https://discuss.huggingface.co/u/AngledLuffa)\
**Post date:** [November 9, 2022, 8:51am UTC](https://discuss.huggingface.co/t/cache-parallelize-long-tokenization-step/25791/1 "2022-11-09T08:51:21Z")

</div>

If I use `run_mlm.py` to train a new model on a large dataset, the cost of tokenization seems quite high. In my particular case, I am looking to use an existing tokenizer associated with the `vinai/phobert-large` model. It takes about 20 minutes to tokenize 1G of text. I have 75G I am hoping to feed to the model, which means about a day to tokenize the whole thing.

Is there a faster way to tokenize the data, or is it at least possible to preprocess the tokenization and distribute the work?

Thanks!

---

<div class="post-metadata">

**Author:** ![AngledLuffa](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@AngledLuffa](https://discuss.huggingface.co/u/AngledLuffa)\
**Post date:** [November 11, 2022, 3:46am UTC](https://discuss.huggingface.co/t/cache-parallelize-long-tokenization-step/25791/2 "2022-11-11T03:46:03Z")

</div>

To answer my own question a bit, it went a lot faster tokenizing when I did

`--preprocessing_num_workers 8`

However, it got stuck in the “Grouping texts in chunks of 256” stage and didn’t do anything for hours:

```auto
Grouping texts in chunks of 256 #1: 100%|███████████████████████████████████████████████████████████| 67063/67063 [1:20:14<00:00, 13.93ba/s]
Grouping texts in chunks of 256 #4: 100%|███████████████████████████████████████████████████████████| 67063/67063 [1:26:42<00:00, 12.89ba/s]
Grouping texts in chunks of 256 #2: 100%|███████████████████████████████████████████████████████████| 67063/67063 [1:28:47<00:00, 12.59ba/s]
Grouping texts in chunks of 256 #2: 97%|█████████████████████████████████████████████████████████▍ | 65285/67063 [1:26:41<02:21, 12.54ba/s]
Grouping texts in chunks of 256 #2: 100%|██████████████████████████████████████████████████████████▉| 67061/67063 [1:28:47<00:00, 16.47ba/s]
Grouping texts in chunks of 256 #4: 100%|██████████████████████████████████████████████████████████▉| 67061/67063 [1:26:41<00:00, 16.50ba/s]

```

After a couple hours of nothing, I gave up and did ctrl-C to restart it, hoping the cached tokenization would be available.

Unfortunately, although I expected the tokenization to be cached at this point, it was not. This is despite there being 200G of stuff in the directory

`../.cache/huggingface/datasets/text/default-54a2ef5fec746c53/0.0.0/acc32f2f2ef863c93c2f30c52f7df6cc9053a1c2230b8d7da0d210404683ca08`

I did a brief search for threads about `run_mlm.py` getting stuck at the `Grouping texts in chunks` phase, and all I found was a thread talking about too many worker processes, which is unfortunate considering one worker process would take 24 hours to tokenize this much stuff.

---

<div class="post-metadata">

**Author:** ![AngledLuffa](https://avatars.discourse-cdn.com/v4/letter/a/e9a140/32.png) [@AngledLuffa](https://discuss.huggingface.co/u/AngledLuffa)\
**Post date:** [November 11, 2022, 3:48am UTC](https://discuss.huggingface.co/t/cache-parallelize-long-tokenization-step/25791/3 "2022-11-11T03:48:37Z")

</div>

The failure of the caching might be explained by this, which is kind of irritating:

````auto
 Parameter 'function'=<function main.<locals>.tokenize_function at 0x7f4bf8d7eef0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.```
````