# T5 fine tuning, loss difference when using labels and decoder\_input\_ids

**URL:** <https://discuss.huggingface.co/t/t5-fine-tuning-loss-difference-when-using-labels-and-decoder-input-ids/1487>\
**Category:** 🤗Transformers\
**Created:** [October 12, 2020, 3:52pm UTC](https://discuss.huggingface.co/t/t5-fine-tuning-loss-difference-when-using-labels-and-decoder-input-ids/1487 "2020-10-12T15:52:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\
**Post date:** [October 12, 2020, 3:52pm UTC](https://discuss.huggingface.co/t/t5-fine-tuning-loss-difference-when-using-labels-and-decoder-input-ids/1487/1 "2020-10-12T15:52:14Z")

</div>

hello,

i am fine tuning T5 for summarization on the news summary dataset.

i am encountering a weird problem when calculating the loss of T5 while training. i get different results when i train computing the loss passing only the ‘labels’ parameter and when i pass both ‘labels’ and ‘decoder\_input\_ids’.  
i find the documentation a bit misleading. in the ‘Training’ section, it says

> the forward function automatically creates the correct decoder\_input\_ids

hereafter you can see the different loss plots i get:

 ![loss](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/24a56ab18055d30d00aede78d17c6501e117f1ec.png)

the green line (copper-rain) is when i do not pass any ‘decoder\_input\_ids’, i.e. in the dataset class

```
def __getitem__ (self, idx):
        item = {k: v[idx] for k, v in self.encodings.items()}
        item['labels'] = self.labels[idx]

```

the blue line (clean-cosmos) is when i do pass the ‘decoder\_input\_ids’ along with the ‘labels’ (following the example in [this notebook](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb)), i.e.

```
def __getitem__ (self, idx):
        item = {k: v[idx] for k, v in self.encodings.items()}
        item['labels'] = self.labels[idx]
        item['decoder_input_ids'] = item['labels']

```

do you kindly have any explanations for this behaviour?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [October 12, 2020, 5:08pm UTC](https://discuss.huggingface.co/t/t5-fine-tuning-loss-difference-when-using-labels-and-decoder-input-ids/1487/2 "2020-10-12T17:08:53Z")

</div>

How are you preparing `labels` ?  
`decoder_input_ids` are shifted to the right and start with `pad` token.  
This is how you should prepare `decoder_input_ids` for T5

```python
decoder_input_ids = labels.new_zeros(labels.shape)
decoder_input_ids[..., 1:] = labels[..., :-1].clone()
decoder_input_ids[..., 0] = self.pad_token_id

```

Consider using `examples/seq2seq` [here](https://github.com/huggingface/transformers/tree/master/examples/seq2seq), for seq2seq experiments, it does lot’s of things for you including this

---

<div class="post-metadata">

**Author:** ![marcoabrate](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marcoabrate/32/441_2.png) [@marcoabrate](https://discuss.huggingface.co/u/marcoabrate)\
**Post date:** [October 12, 2020, 5:45pm UTC](https://discuss.huggingface.co/t/t5-fine-tuning-loss-difference-when-using-labels-and-decoder-input-ids/1487/3 "2020-10-12T17:45:46Z")

</div>

thank you for your help!

i see. indeed, now i get exactly the same results as by passing ‘labels’ only (since i set the same seed). i was biased by the fact that i was getting a much lower loss passing the labels to the decode\_input\_ids as is. do you have any clue why i get this behaviour?

i wonder decode\_input\_ids is not useful at all when passing labels to the trained model. i don’t really understand why in the notebook i linked they give it as a parameter.