# How does T5 create the correct decoder\_input\_ids?

**URL:** <https://discuss.huggingface.co/t/how-does-t5-create-the-correct-decoder-input-ids/1201>\
**Category:** 🤗Transformers\
**Created:** [September 19, 2020, 8:43pm UTC](https://discuss.huggingface.co/t/how-does-t5-create-the-correct-decoder-input-ids/1201 "2020-09-19T20:43:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lintang](https://avatars.discourse-cdn.com/v4/letter/l/f14d63/32.png) [@lintang](https://discuss.huggingface.co/u/lintang)\
**Post date:** [September 19, 2020, 8:43pm UTC](https://discuss.huggingface.co/t/how-does-t5-create-the-correct-decoder-input-ids/1201/1 "2020-09-19T20:43:26Z")

</div>

> **[T5 — transformers 3.1.0 documentation](https://huggingface.co/transformers/model_doc/t5.html#training)**

I have been reading the documentation for the T5 model and in the training section, for both unsupervised denoising and supervised training, the comment states the model is able to create the correct `decoder_input_ids`. However, I’m not sure how the model is able to do. The documentation tells me that if the `decoder_input_ids` is None (which it is by default) it takes the values of `input_ids`. Does this mean the start token is the first token of the input sequence?

The description also mentions a start-sequence token. But I can’t seem to find where this is generated or what the token is.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 20, 2020, 12:14pm UTC](https://discuss.huggingface.co/t/how-does-t5-create-the-correct-decoder-input-ids/1201/2 "2020-09-20T12:14:22Z")

</div>

The `T5PreTrainedModel._shift_right` method [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_t5.py#L622) takes `labels` and prepares `decoder_input_ids`

Basically, all it does is,

takes `labels` -\> add pad token at the beginning -\> remove the `eos` -\> `decoder_input_ids`

---

<div class="post-metadata">

**Author:** ![lintang](https://avatars.discourse-cdn.com/v4/letter/l/f14d63/32.png) [@lintang](https://discuss.huggingface.co/u/lintang)\
**Post date:** [September 21, 2020, 1:30pm UTC](https://discuss.huggingface.co/t/how-does-t5-create-the-correct-decoder-input-ids/1201/3 "2020-09-21T13:30:40Z")

</div>

Thanks for pointing this out for me!