# A question about the DataCollator for LM

**URL:** <https://discuss.huggingface.co/t/a-question-about-the-datacollator-for-lm/85273>\
**Category:** 🤗Tokenizers\
**Created:** [May 5, 2024, 3:34pm UTC](https://discuss.huggingface.co/t/a-question-about-the-datacollator-for-lm/85273 "2024-05-05T15:34:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dydavide](https://avatars.discourse-cdn.com/v4/letter/d/22d042/32.png) [@dydavide](https://discuss.huggingface.co/u/dydavide)\
**Post date:** [May 5, 2024, 3:34pm UTC](https://discuss.huggingface.co/t/a-question-about-the-datacollator-for-lm/85273/1 "2024-05-05T15:34:01Z")

</div>

I have a question about DataCollatorForLanguageModeling, when training a LM.  
I saw [this video](https://youtu.be/98jROp7Ij9A?si=-KZm0JB-Z1aXb5Nc), which explains very well, how the training process works.  
Starting from minute 5:12 it says _“the datacollator shifts the input, such that the label is the next token in the sequence for every single token in the input”_  
It make sense to me and it is a nice explanation about what is happening behind the scene.  
But then, looking into the [documentation](https://huggingface.co/docs/transformers/main_classes/data_collator#transformers.DataCollatorForLanguageModeling) for understanding the _mlm_ parameter I found the following:

**mlm** (`bool`, _optional_, defaults to `True`) — Whether or not to use masked language modeling. If set to `False`, the labels are the same as the inputs with the padding tokens ignored (by setting them to -100). Otherwise, the labels are -100 for non-masked tokens and the value to predict for the masked token.

So, now I’m totally confused. Is DataCollator shifting the tokens to the left? Or it is controlling only the behavior of the padding tokens?

Thanks

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [May 6, 2024, 9:06am UTC](https://discuss.huggingface.co/t/a-question-about-the-datacollator-for-lm/85273/2 "2024-05-06T09:06:30Z")

</div>

Hi,

No the data collator itself is not shifting any tokens. Typically the labels are just a copy of the input\_ids, with padding tokens replaced by -100 (which is the ignore index of the cross-entropy loss in PyTorch).

The shifting of the tokens one position happens inside the model (hence the user doesn’t need to take care of that). This can be seen [here](https://github.com/huggingface/transformers/blob/09edd77f64481cc9ce03067f4cfe74cef9d5d776/src/transformers/models/llama/modeling_llama.py#L1228-L1238) for Llama for instance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [May 6, 2024, 9:06pm UTC](https://discuss.huggingface.co/t/a-question-about-the-datacollator-for-lm/85273/3 "2024-05-06T21:06:37Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.