# Tokenizer dataset is very slow

**URL:** <https://discuss.huggingface.co/t/tokenizer-dataset-is-very-slow/19722>\
**Category:** 🤗Tokenizers\
**Created:** [June 28, 2022, 12:35pm UTC](https://discuss.huggingface.co/t/tokenizer-dataset-is-very-slow/19722 "2022-06-28T12:35:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ccfeidao](https://avatars.discourse-cdn.com/v4/letter/c/f08c70/32.png) [@ccfeidao](https://discuss.huggingface.co/u/ccfeidao)\
**Post date:** [June 28, 2022, 12:35pm UTC](https://discuss.huggingface.co/t/tokenizer-dataset-is-very-slow/19722/1 "2022-06-28T12:35:02Z")

</div>

This is my tokenizer method. I found that no matter how much batch\_size is set, the speed is the same. Tokenizer Spend time even longer than training. How cloud I do. Thanks very much.

```auto
def tokenize_function(example):
    return tokenizer(example["sentence1"], truncation=True, max_length = 512)
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True, batch_size = 8)
tokenized_datasets = tokenized_datasets.remove_columns(["sentence1"])

```

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [June 28, 2022, 6:14pm UTC](https://discuss.huggingface.co/t/tokenizer-dataset-is-very-slow/19722/2 "2022-06-28T18:14:48Z")

</div>

Hi! What tokenizer are you using? What does `tokenizer.is_fast` return? If the returned value is `False`, you can set `num_proc` \> 1 to leverage multiprocessing in `map`. Fast tokenizers use multithreading to process a batch in parallel on a single process by default, so it doesn’t make sense to use `num_proc` there.

---

<div class="post-metadata">

**Author:** ![msinghy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/msinghy/32/24386_2.png) [@msinghy](https://discuss.huggingface.co/u/msinghy)\
**Post date:** [March 2, 2024, 6:19am UTC](https://discuss.huggingface.co/t/tokenizer-dataset-is-very-slow/19722/4 "2024-03-02T06:19:01Z")

</div>

I’ve checked mine, and I have a fast tokenizer. However, it’s still taking about 20 seconds per example for tokenization, which is too slow.

Here’s the code,

```auto
base_model_id = "google/gemma-7b"
tokenizer = AutoTokenizer.from_pretrained(
    base_model_id,
    padding_side="left",
    add_eos_token=True,
    add_bos_token=True,
)
tokenizer.pad_token = tokenizer.eos_token

```

```auto
max_length = 1026

def generate_and_tokenize_prompt(prompt):
    result = tokenizer(
        formatting_func(prompt),
        truncation=True,
        max_length=max_length,
        padding="max_length",
    )
    result["labels"] = result["input_ids"].copy()
    return result

train_dataset = dataset_split['train']
eval_dataset = dataset_split['test']
tokenized_train_dataset = train_dataset.map(generate_and_tokenize_prompt)
tokenized_val_dataset = eval_dataset.map(generate_and_tokenize_prompt)

```

Can someone please help me figure out what I’m missing? Thanks.

---

<div class="post-metadata">

**Author:** ![mariosasko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mariosasko/32/31548_2.png) [@mariosasko](https://discuss.huggingface.co/u/mariosasko)\
**Post date:** [March 2, 2024, 3:47pm UTC](https://discuss.huggingface.co/t/tokenizer-dataset-is-very-slow/19722/5 "2024-03-02T15:47:07Z")

</div>

Feel free to report this issue in the [`tokenizers`](https://github.com/huggingface/tokenizers) repo.

---

<div class="post-metadata">

**Author:** ![hamingsi](https://avatars.discourse-cdn.com/v4/letter/h/eb8c5e/32.png) [@hamingsi](https://discuss.huggingface.co/u/hamingsi)\
**Post date:** [November 25, 2025, 9:30am UTC](https://discuss.huggingface.co/t/tokenizer-dataset-is-very-slow/19722/6 "2025-11-25T09:30:45Z")

</div>

You may need to set the preprocessing\_num\_workers as None like:

```auto
accelerate launch \
    --num_processes 8 \
    --num_machines 1 \
    --main_process_port 44144 \
    --mixed_precision fp16 \
    --dynamo_backend no \
    pretrain.py \
    --dataset_name \
        /data/***/datasets/STORIES \
    --model_name_or_path bert-base-uncased \
    --per_device_train_batch_size 64 \
    --per_device_eval_batch_size 64 \
    --learning_rate 2e-4 \
    --max_train_steps 500000 \
    --num_warmup_steps 5000 \
    --output_dir ./base \
    --max_seq_length 128 \
    --checkpointing_steps 50000 \
    # --preprocessing_num_workers None \
    --with_tracking \
    --report_to wandb

```

Result in:

```auto
Running tokenizer on every text in dataset: 95%|██████████████████████████████████████████████████████████████████████████████████████ | 77660856/82086247 [1:09:52<6:53:00, 178.58 examples/s]

```

For me, I just set the `preprocessing_num_workers 16` at the beginning and remove the `preprocessing_num_workers` apporaching the end. This will result in almost 20000 examples/s at the beginning and 180 examples/s at the end, but still better then 2 examples/s. However, I still don’t understand why single process will run faster.