# When to use a DataCollator for SFTTrainer

**URL:** <https://discuss.huggingface.co/t/when-to-use-a-datacollator-for-sfttrainer/79696>\
**Category:** Beginners\
**Created:** [April 1, 2024, 5:02pm UTC](https://discuss.huggingface.co/t/when-to-use-a-datacollator-for-sfttrainer/79696 "2024-04-01T17:02:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![AdL8](https://avatars.discourse-cdn.com/v4/letter/a/3be4f8/32.png) [@AdL8](https://discuss.huggingface.co/u/AdL8)\
**Post date:** [April 1, 2024, 5:02pm UTC](https://discuss.huggingface.co/t/when-to-use-a-datacollator-for-sfttrainer/79696/1 "2024-04-01T17:02:49Z")

</div>

Hello,

In the SFTTrainer document, it is stated that if the dataset is in the right format, we dont need to specify a DataCollator with a response\_template.

However, after I formatted my dataset with the TinyLlama/TinyLlama-1.1B-Chat-v1.0 tokenizer.apply\_chat\_template, the labels are not correct in the train dataloader.

Here is a sample from the dataloader of the SFTTrainer:

( I removed the \<(s)\> )  
Input : \<|user|\>Which is bigger, the moon or the sun?'\<|assistant|\> The sun.  
Inputs\_ids : tensor([ 1, 529, 29989, 1792, 29989, 29958, 13, 8809, 436, 338,  
16600, 29892, 278, 18786, 470, 278, 6575, 29973, 2, 29871,  
13, 29966, 29989, 465, 22137, 29989, 29958, 13, 1576, 6575,  
29889, 2, 29871, 13, 2, 2, 2, 2])

Labels : tensor([ 1, 529, 29989, 1792, 29989, 29958, 13, 8809, 436, 338,  
16600, 29892, 278, 18786, 470, 278, 6575, 29973, -100, 29871,  
13, 29966, 29989, 465, 22137, 29989, 29958, 13, 1576, 6575,  
29889, -100, 29871, 13, -100, -100, -100, -100])

Is the labels supposed to be like that ? Or is it incorrect ?

---

<div class="post-metadata">

**Author:** ![marches7](https://avatars.discourse-cdn.com/v4/letter/m/0ea827/32.png) [@marches7](https://discuss.huggingface.co/u/marches7)\
**Post date:** [March 15, 2025, 11:52am UTC](https://discuss.huggingface.co/t/when-to-use-a-datacollator-for-sfttrainer/79696/2 "2025-03-15T11:52:54Z")

</div>

Well, I think your ‘Labels’ are wrong. Since Llama is a decoder-only transformers model, it uses the previous tokens to predict the next token, your ‘Labels’ should only mask the instruction’s input\_ids. That means the first few tokens of your ‘Labels’ should be -100.