# Speeding up Tokenization on large text corpus

**URL:** <https://discuss.huggingface.co/t/speeding-up-tokenization-on-large-text-corpus/23610>\
**Category:** 🤗Transformers\
**Created:** [September 26, 2022, 5:39am UTC](https://discuss.huggingface.co/t/speeding-up-tokenization-on-large-text-corpus/23610 "2022-09-26T05:39:15Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jaidev](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jaidev](https://discuss.huggingface.co/u/Jaidev)\
**Post date:** [September 26, 2022, 5:39am UTC](https://discuss.huggingface.co/t/speeding-up-tokenization-on-large-text-corpus/23610/1 "2022-09-26T05:39:15Z")

</div>

I have a similar issue, using a pretrained WordPiece tokenizer on a large corpus of text takes several hours. I’m doing:

tokenizer = AutoTokenizer.from\_pretrained(“distilbert-base-uncased”)  
train\_tokenized\_encodings = tokenizer(df[df.split==‘train’].text.tolist(), truncation=True, padding=True, max\_length=MAX\_LENGTH)

Any suggestions for speed up?

Is there a way to parallelize this? (Or does the above automatically use multiple workers?)