# Understanding T5 with custom embedding

**URL:** <https://discuss.huggingface.co/t/understanding-t5-with-custom-embedding/162325>\
**Category:** 🤗Transformers\
**Created:** [July 9, 2025, 8:27am UTC](https://discuss.huggingface.co/t/understanding-t5-with-custom-embedding/162325 "2025-07-09T08:27:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jannik97](https://avatars.discourse-cdn.com/v4/letter/j/51bf81/32.png) [@Jannik97](https://discuss.huggingface.co/u/Jannik97)\
**Post date:** [July 9, 2025, 8:27am UTC](https://discuss.huggingface.co/t/understanding-t5-with-custom-embedding/162325/1 "2025-07-09T08:27:05Z")

</div>

Hello everyone,

I am passing a custom input embedding via inputs\_embeds to a T5ForConditionalGeneration model like this:

```auto
encoder_outputs = model.encoder(
        inputs_embeds=input_embedding_batch,
        attention_mask=attention_mask
    )

```

Which gets me a shape of [5, 122, 1024]

Then I pass this to generate my target like this:

```auto
target = model.generate(
    encoder_outputs=encoder_outputs,
    attention_mask=attention_mask,
    max_length=max_len,
    min_length=min_len,
    **gen_kwargs_fold2AA
)

```

Normally (when I pass just the token\_ids) the model would generate a sequence of the same length, so 122 in my case. However here it always generates a sequence of length max\_len and I don’t know why. Is the encoder\_output still given as starting sequence and the decoder just continues generating based on this ?

Thank you for your time and help

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 9, 2025, 9:56am UTC](https://discuss.huggingface.co/t/understanding-t5-with-custom-embedding/162325/2 "2025-07-09T09:56:40Z")

</div>

This is the output I got in my environment…

```py
import torch
from transformers import T5ForConditionalGeneration, T5Tokenizer

# Load model and tokenizer
device = "cuda" if torch.cuda.is_available() else "cpu"
model = T5ForConditionalGeneration.from_pretrained("t5-small").to(device)
tokenizer = T5Tokenizer.from_pretrained("t5-small", lagacy=False)

# Prepare a batch of input token IDs and attention mask
text = "translate English to German: The weather is nice today."
batch = tokenizer(text, return_tensors="pt").to(device)
input_ids = batch.input_ids # shape: [1, max_len]
attention_mask = batch.attention_mask
max_len = 4
min_len = 2

# Manually compute input embeddings via the encoder's embedding layer
inputs_embeds = model.get_encoder().embed_tokens(input_ids)
print(f"inputs_embeds shape: {inputs_embeds.shape}")
# inputs_embeds shape: [1, max_len, d_model]

# 1) Generate using precomputed encoder_outputs
encoder_outputs = model.encoder(
    inputs_embeds=inputs_embeds,
    attention_mask=attention_mask
)
print(f"encoder_outputs shape: {encoder_outputs.last_hidden_state.shape}")
# Decoder sees only a BOS token as prefix, so length = max_length
generated_ids1 = model.generate(
    encoder_outputs=encoder_outputs,
    attention_mask=attention_mask,
    max_length=max_len, # we might expect max_len, but decoder prefix is length 1
    min_length=min_len,
)
print("\nUsing encoder_outputs:")
print("Requested max_length =", max_len)
print("Generated length =", generated_ids1.shape[-1])
print(tokenizer.decode(generated_ids1[0], skip_special_tokens=True))

# 2) Generate by passing embeddings directly to generate()
# Here generate() handles the encoder internally and accounts for prompt length
generated_ids2 = model.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=attention_mask,
    max_length=max_len, # now prompt length is max_len, so output length matches
    min_length=min_len,
)
print("\nUsing inputs_embeds in generate:")
print("Requested max_length =", max_len)
print("Generated length =", generated_ids2.shape[-1])
print(tokenizer.decode(generated_ids2[0], skip_special_tokens=True))

"""
inputs_embeds shape: torch.Size([1, 12, 512])
encoder_outputs shape: torch.Size([1, 12, 512])

Using encoder_outputs:
Requested max_length = 4
Generated length = 4
Das Wetter ist

Using inputs_embeds in generate:
Requested max_length = 4
Generated length = 4
Das Wetter ist
"""

```

---

<div class="post-metadata">

**Author:** ![Amber14L](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/amber14l/32/50832_2.png) [@Amber14L](https://discuss.huggingface.co/u/Amber14L)\
**Post date:** [July 9, 2025, 10:00am UTC](https://discuss.huggingface.co/t/understanding-t5-with-custom-embedding/162325/3 "2025-07-09T10:00:00Z")

</div>

When you pass `encoder_outputs` directly, `generate()` doesn’t know your input length or tokens so it just generates until `max_length` or EOS. It’s not continuing from anything, just starting fresh based on the encoder output.

---

<div class="post-metadata">

**Author:** ![Jannik97](https://avatars.discourse-cdn.com/v4/letter/j/51bf81/32.png) [@Jannik97](https://discuss.huggingface.co/u/Jannik97)\
**Post date:** [July 9, 2025, 1:50pm UTC](https://discuss.huggingface.co/t/understanding-t5-with-custom-embedding/162325/4 "2025-07-09T13:50:07Z")

</div>

Hey,

Thank you very much for your answers. Now I understand. So I can just provide the encoder\_outputs to the model.generate() and set the max\_length to my seq\_len of my encoder\_output