# Pre-tokenization vs. mini-batch tokenization and TOKENIZERS\_PARALLELISM warning

**URL:** <https://discuss.huggingface.co/t/pre-tokenization-vs-mini-batch-tokenization-and-tokenizers-parallelism-warning/11387>\
**Category:** 🤗Transformers\
**Created:** [November 5, 2021, 1:12pm UTC](https://discuss.huggingface.co/t/pre-tokenization-vs-mini-batch-tokenization-and-tokenizers-parallelism-warning/11387 "2021-11-05T13:12:42Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dpernes](https://avatars.discourse-cdn.com/v4/letter/d/e36b37/32.png) [@dpernes](https://discuss.huggingface.co/u/dpernes)\
**Post date:** [November 5, 2021, 1:12pm UTC](https://discuss.huggingface.co/t/pre-tokenization-vs-mini-batch-tokenization-and-tokenizers-parallelism-warning/11387/1 "2021-11-05T13:12:42Z")

</div>

I am using BART and its `BartTokenizeFast` for a Seq2Seq application. Since my dataset is fixed (i.e., I’m not using any kind of data augmentation or transformation during the training process), I thought that the most sensible option would be:

- Tokenizing all the sequences in the dataset in a preprocessing step, without padding.
- Padding the sequences online as needed for each mini-batch, using `DataCollatorForSeq2Seq` as the dataloader `collate_fn`.

However, when I do this, I get the warning:

```auto
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
        - Avoid using `tokenizers` before the fork if possible
        - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

```

I understand that this warning happens because I call the `tokenizer` before the multiprocessing starts, i.e. before iterating through the dataloader. My questions are:

1. Am I wrong in assuming that pre-tokenization is, in my case, a better option than tokenizing each mini-batch in the dataloader?

2. Is it safe to ignore the warning in this situation?

3. Should I set TOKENIZERS\_PARALLELISM=true, or TOKENIZERS\_PARALLELISM=false, or either option doesn’t make any difference in my case?

---

<div class="post-metadata">

**Author:** ![Sifal](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@Sifal](https://discuss.huggingface.co/u/Sifal)\
**Post date:** [October 28, 2023, 8:31am UTC](https://discuss.huggingface.co/t/pre-tokenization-vs-mini-batch-tokenization-and-tokenizers-parallelism-warning/11387/2 "2023-10-28T08:31:40Z")

</div>

I’m running into the same wall, do you have a recomended approach?

---

<div class="post-metadata">

**Author:** ![Iamexperimenting](https://avatars.discourse-cdn.com/v4/letter/i/839c29/32.png) [@Iamexperimenting](https://discuss.huggingface.co/u/Iamexperimenting)\
**Post date:** [March 3, 2024, 2:36pm UTC](https://discuss.huggingface.co/t/pre-tokenization-vs-mini-batch-tokenization-and-tokenizers-parallelism-warning/11387/3 "2024-03-03T14:36:37Z")

</div>

@nielsr I’m also facing the same problem. Can you guide here?