# DataCollator vs. Tokenizers

**URL:** <https://discuss.huggingface.co/t/datacollator-vs-tokenizers/5897>\
**Category:** 🤗Transformers\
**Created:** [May 1, 2021, 12:40am UTC](https://discuss.huggingface.co/t/datacollator-vs-tokenizers/5897 "2021-05-01T00:40:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![hamel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/hamel/32/1559_2.png) [@hamel](https://discuss.huggingface.co/u/hamel)\
**Post date:** [May 1, 2021, 12:40am UTC](https://discuss.huggingface.co/t/datacollator-vs-tokenizers/5897/1 "2021-05-01T00:40:01Z")

</div>

In @sgugger 's example notebook [Token Classification](https://github.com/huggingface/notebooks/blob/master/examples/token_classification.ipynb)

A `DataCollatorForTokenClassification` is used:

```auto
from transformers import DataCollatorForTokenClassification
data_collator = DataCollatorForTokenClassification(tokenizer)
...
trainer = Trainer(
    model,
    args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["validation"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)

```

I am trying to figure out what the real purpose of this is. It appears that the purpose of `DataCollatorForTokenClassification` is for padding, truncation, etc. But you can also do this in the tokenizer. Why do we need this extra thing, then? Is it because DataCollator does it per batch instead on the fly and is more efficient?

Thanks.

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [May 1, 2021, 11:15am UTC](https://discuss.huggingface.co/t/datacollator-vs-tokenizers/5897/2 "2021-05-01T11:15:10Z")

</div>

hey @hamel, welcome to the 🤗 forum!

you’re spot on about using data collators to do padding on-the-fly. to understand why this helps, consider the following scenarios:

1. use the tokenizer to pad each example in the dataset to the length of the longest example in the dataset
2. use the tokenizer and `DataCollatorWithPadding` ([docs](https://huggingface.co/transformers/master/main_classes/data_collator.html?highlight=datacollator#datacollatorwithpadding)) to pad each example in a batch to the length of the longest example in the batch

clearly, scenario 2 is more efficient, especially in cases where a few examples happen to be much longer than the median length and scenario 1 would introduce a lot of unnecessary padding.

btw under the hood, the data collators are fed to the `collate_fn` argument of pytorch’s `DataLoader`, see e.g. here: [transformers/trainer.py at 4e7bf94e7280d2b725ac4644dbe9808560afa5d8 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/4e7bf94e7280d2b725ac4644dbe9808560afa5d8/src/transformers/trainer.py#L617)

the pytorch [docs](https://pytorch.org/docs/stable/data.html#working-with-collate-fn) are not super informative on `collate_fn` itself, but you can find various discussion in their forums (e.g. [here](https://discuss.pytorch.org/t/how-to-create-batches-of-a-list-of-varying-dimension-tensors/50773/14))