# T5 tokenizer's post-processor is suboptimal for truncated sequences for seq2seq finetuning

**URL:** <https://discuss.huggingface.co/t/t5-tokenizers-post-processor-is-suboptimal-for-truncated-sequences-for-seq2seq-finetuning/45760>\
**Category:** 🤗Transformers\
**Created:** [July 5, 2023, 3:25pm UTC](https://discuss.huggingface.co/t/t5-tokenizers-post-processor-is-suboptimal-for-truncated-sequences-for-seq2seq-finetuning/45760 "2023-07-05T15:25:30Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mmlynarik](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@mmlynarik](https://discuss.huggingface.co/u/mmlynarik)\
**Post date:** [July 5, 2023, 3:25pm UTC](https://discuss.huggingface.co/t/t5-tokenizers-post-processor-is-suboptimal-for-truncated-sequences-for-seq2seq-finetuning/45760/1 "2023-07-05T15:25:30Z")

</div>

Hello,

I’m not sure if this is a bug, a feature, or potential known tokenizers/transformers library limitation that can’t be avoided, but I would like to highlight that when T5 model and its tokenizer is used for preparing inputs for summarization model fine-tuning, it uses suboptimal encoding scheme.

Concretely, **when the tokenizer is set to apply truncation and the input text sequence is longer than `max_length`** (i.e. the tokenizer truncates it), the produced labels tensor always contain EOS token (here eos\_token\_id = 1). This behaviour in fact should not be present because the sequence has not been ended properly, just truncated, and as a result, model learns to generate EOS token incorrectly also in cases when it should not to.

If the input sequence is shorter than `max_length`, the problem does not occur because **in this case, the labels should contain eos token,** however, the generated `decoder_input_ids` tensor (using `prepare_decoder_input_ids_from_labels()`) contains EOS token, which also should not be present, although this is not a serious problem as the loss computation excludes this token/timestep from consideration due to its label being pad token-ed.

My question is: should not be mentioned in the documentation, that this kind of limitation is present in the model? In my opinion, this is a tradeoff for performance purposes, because the proper handling of the EOS token would require “conditional tokenization logic”, meaning if the sequence is longer than max\_length, do not apply EOS token, which is currently not supported by tokenizers library and I think would not be in the near future.

```auto
    from transformers import T5TokenizerFast, T5ForConditionalGeneration
    model:T5ForConditionalGeneration = T5ForConditionalGeneration.from_pretrained("t5-small")
    tokenizer: T5TokenizerFast = T5TokenizerFast.from_pretrained("t5-small")

    text = "Here is a short article."
    text_2 = "Here is a long article, which exceeds model max length."
    texts = [text, text_2]

    inputs = tokenizer(texts, padding=True, truncation=True, max_length=10, return_tensors="pt")
    labels = inputs["input_ids"]
    decoder_input_ids = model.prepare_decoder_input_ids_from_labels(labels)

    print(labels)
    print(decoder_input_ids)

```

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/7/9/79ec7b6d8f60286968d60b66a3b89887d76d9bee.png)