# Tokenizer progress bar

**URL:** <https://discuss.huggingface.co/t/tokenizer-progress-bar/1147>\
**Category:** 🤗Transformers\
**Created:** [September 15, 2020, 3:51pm UTC](https://discuss.huggingface.co/t/tokenizer-progress-bar/1147 "2020-09-15T15:51:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![congcongwang](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/congcongwang/32/3282_2.png) [@congcongwang](https://discuss.huggingface.co/u/congcongwang)\
**Post date:** [September 15, 2020, 3:51pm UTC](https://discuss.huggingface.co/t/tokenizer-progress-bar/1147/1 "2020-09-15T15:51:29Z")

</div>

Hi Great Community,

Is it possible to have a progress bar to track the tokenisation process when calling the following method?

tokenizer(large\_batch, padding=True, truncation=True,max\_length=512)

---

<div class="post-metadata">

**Author:** ![lhallee](https://avatars.discourse-cdn.com/v4/letter/l/dfb087/32.png) [@lhallee](https://discuss.huggingface.co/u/lhallee)\
**Post date:** [April 6, 2023, 7:18pm UTC](https://discuss.huggingface.co/t/tokenizer-progress-bar/1147/2 "2023-04-06T19:18:01Z")

</div>

I would also like a progress bar for tokenizing! Maybe a verbose setting? Did you ever hear back about this?

---

<div class="post-metadata">

**Author:** ![hughiephan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hughiephan/32/28723_2.png) [@hughiephan](https://discuss.huggingface.co/u/hughiephan)\
**Post date:** [August 6, 2023, 1:52pm UTC](https://discuss.huggingface.co/t/tokenizer-progress-bar/1147/3 "2023-08-06T13:52:08Z")

</div>

tqdm library allows you to output a progress bar for the tokenizer.

```auto
from tqdm import tqdm
def tokenizer_with_progress(large_batch):
    tokenized_texts = []
    for text in tqdm(large_batch, desc="Tokenizing", unit="text"):
        tokenized_texts.append(tokenizer(text, padding=True, truncation=True, max_length=512))
    return tokenized_texts

train_encodings = tokenizer_with_progress(training_sentences)
test_encodings = tokenizer_with_progress(testing_sentences)

```