# Speed up tokenizer training

**URL:** <https://discuss.huggingface.co/t/speed-up-tokenizer-training/76417>\
**Category:** 🤗Tokenizers\
**Created:** [March 7, 2024, 4:28pm UTC](https://discuss.huggingface.co/t/speed-up-tokenizer-training/76417 "2024-03-07T16:28:24Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![astein0](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@astein0](https://discuss.huggingface.co/u/astein0)\
**Post date:** [March 7, 2024, 4:28pm UTC](https://discuss.huggingface.co/t/speed-up-tokenizer-training/76417/1 "2024-03-07T16:28:24Z")

</div>

I am training a tokenizer from scratch on a large dataset. Is there a way to take advantage of parallelism in the tokenizer trainer?

---

<div class="post-metadata">

**Author:** ![alisawuffles](https://avatars.discourse-cdn.com/v4/letter/a/c68b51/32.png) [@alisawuffles](https://discuss.huggingface.co/u/alisawuffles)\
**Post date:** [April 22, 2024, 8:07pm UTC](https://discuss.huggingface.co/t/speed-up-tokenizer-training/76417/2 "2024-04-22T20:07:43Z")

</div>

I also have this question! I am training several tokenizers from scratch for a research project, and it is quite time-consuming — ~15 hours to count the pairs for 4.5G of text. What resources would be most helpful to request for a training job? Is there a way to take advantage of parallelism, as @astein0 asks?

---

<div class="post-metadata">

**Author:** ![maveriq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/maveriq/32/27075_2.png) [@maveriq](https://discuss.huggingface.co/u/maveriq)\
**Post date:** [May 17, 2024, 11:47pm UTC](https://discuss.huggingface.co/t/speed-up-tokenizer-training/76417/3 "2024-05-17T23:47:45Z")

</div>

same question. I have come up with a pyspark implementation of [minbpe](https://github.com/karpathy/minbpe/tree/master) which will give you merges and vocabulary for a BPE tokenizer. It still needs some more work to optimize the speed but it works decent. Ping me if anyone wants to collaborate on that.

Meanwhile it would be nice to have an official implementation, either using tokenizer or the datatrove library. @lhoestq

---

<div class="post-metadata">

**Author:** ![alisawuffles](https://avatars.discourse-cdn.com/v4/letter/a/c68b51/32.png) [@alisawuffles](https://discuss.huggingface.co/u/alisawuffles)\
**Post date:** [May 28, 2024, 8:01pm UTC](https://discuss.huggingface.co/t/speed-up-tokenizer-training/76417/4 "2024-05-28T20:01:40Z")

</div>

It turns out that setting `TOKENIZERS_PARALLELISM=true` solved my problem. 🙂

---

<div class="post-metadata">

**Author:** ![subatomicseer](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@subatomicseer](https://discuss.huggingface.co/u/subatomicseer)\
**Post date:** [September 17, 2024, 1:28pm UTC](https://discuss.huggingface.co/t/speed-up-tokenizer-training/76417/5 "2024-09-17T13:28:59Z")

</div>

If you are using any form of multiprocessing after importing the tokenisers packages, then it will set the parallelism off.

So doing something like this will ensure that the tokenizers will use parallelism:

```auto
dataset = dataset.map(map_fn, batched=True, batch_size=64, num_proc=128)

# set the following environment variable to true and then import tokenizer modules
os.environ["TOKENIZERS_PARALLELISM"] = "true"
from tokenizers import (
  decoders,
  models,
  pre_tokenizers,
  normalizers,
  trainers,
  Tokenizer,
  Regex
)
tokenizer = Tokenizer(models.Unigram())

```

It worked for me.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [December 3, 2024, 11:00pm UTC](https://discuss.huggingface.co/t/speed-up-tokenizer-training/76417/6 "2024-12-03T23:00:40Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.