# Error when trying to visualize attention in T5 model

**URL:** <https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350>\
**Category:** Beginners\
**Created:** [April 2, 2023, 8:00pm UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350 "2023-04-02T20:00:30Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![snork-maiden](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@snork-maiden](https://discuss.huggingface.co/u/snork-maiden)\
**Post date:** [April 2, 2023, 8:00pm UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/1 "2023-04-02T20:00:30Z")

</div>

I have a pretrained T5 model that predicts the solution to quadratic equations and I want to use _bertviz_ library for visualizing attention. In all the examples I found the length of the input and output are the same, but in my case they are different.

```auto
tokenizer = PreTrainedTokenizerFast.from_pretrained("my_repo/content")
model = T5ForConditionalGeneration.from_pretrained("my_repo/content", output_attentions=True)

```

For example for this input:

```auto
inputs = tokenizer("7*x^2+3556*x+451612=0", return_tensors="pt")

```

The model predicts:

```auto
outputs = model.generate(inputs.input_ids, attention_mask=inputs.attention_mask, max_length=80, min_length=10, output_attentions=True, return_dict_in_generate=True)

```

This sequnce:

```auto
D = 3556 ^ 2 - 4 * 7 * 4 5 1 6 1 2 = 2 1 ; x 1 = ( - 3556 + ( 2 1 ) * * 0. 5 ) / / ( 2 * 7 ) = - 2. 0 ; x 2 = ( - 3556 - ( 2 1 ) * * 0. 5 ) / / ( 2 * 7 ) = - 2. 0

```

It’s length is 79, whereas the length of the input is 18.

Then I do as in [the example](https://github.com/jessevig/bertviz#encoder-decoder-models-bart-t5-etc):

```auto
encoder_input_ids = tokenizer("7*x^2+3556*x+451612=0", return_tensors="pt", add_special_tokens=True).input_ids

with tokenizer.as_target_tokenizer():
    decoder_input_ids = tokenizer("D = 3556 ^ 2 - 4 * 7 * 4 5 1 6 1 2 = 2 1 ; x 1 = ( - 3556 + ( 2 1 ) * * 0. 5 ) / / ( 2 * 7 ) = - 2. 0 ; x 2 = ( - 3556 - ( 2 1 ) * * 0. 5 ) / / ( 2 * 7 ) = - 2. 0", 
                                  return_tensors="pt", add_special_tokens=True).input_ids

encoder_text = tokenizer.convert_ids_to_tokens(encoder_input_ids[0])
decoder_text = tokenizer.convert_ids_to_tokens(decoder_input_ids[0])

model_view(
    cross_attention = outputs.cross_attentions[0],
    encoder_attention = encoder_attention, 
    decoder_attention = decoder_attention,
    encoder_tokens = encoder_text,
    decoder_tokens = decoder_text)

```

However I get this error:

```auto
AttributeError: 'tuple' object has no attribute 'shape'

```

For some reason the attentions I get are in a form of tuples (cross attention is even a tuple of tuples). In bertviz docs it is said that the dimensions should be like this:

```auto
For encoder-decoder models:
                encoder_attention: list of ``torch.FloatTensor``(one for each layer) of shape
                    ``(batch_size(must be 1), num_heads, encoder_sequence_length, encoder_sequence_length)``
                decoder_attention: list of ``torch.FloatTensor``(one for each layer) of shape
                    ``(batch_size(must be 1), num_heads, decoder_sequence_length, decoder_sequence_length)``
                cross_attention: list of ``torch.FloatTensor``(one for each layer) of shape
                    ``(batch_size(must be 1), num_heads, decoder_sequence_length, encoder_sequence_length)``
                encoder_tokens: list of tokens for encoder input
                decoder_tokens: list of tokens for decoder input

```

I don’t understand why am I not getting the right dimensions. Is it because my input and ouput sequences are of different sizes?

I trained my model on a custom dataset. I changed it’s `lm_head.out_features` to `1` and it’s `vocab_size` to `100_000`.

Some info about dimensions:

```auto
cross_attention = outputs.cross_attentions
len(cross_attention[0]) = 12
cross_attention[0][0].shape = torch.Size([1, 12, 1, 18])

decoder_attention = outputs.decoder_attentions
len(decoder_attention[0]) = 12
decoder_attention[0][0].shape = torch.Size([1, 12, 1, 1])

encoder_attention = outputs.encoder_attentions
len(encoder_attention) = 12
encoder_attention[0].shape = torch.Size([1, 12, 18, 18])

```

If I do something like this:

```auto
model_view(
    cross_attention = outputs.cross_attentions[0],
    encoder_attention = encoder_attention[0], 
    decoder_attention = decoder_attention[0],
    encoder_tokens = encoder_text,
    decoder_tokens = decoder_text)

```

I get this error:

```auto
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-53-ae747cb8eba0> in <cell line: 1>()
----> 1 model_view(
      2 cross_attention = outputs.cross_attentions[0],
      3 encoder_attention = encoder_attention[0],
      4 decoder_attention = decoder_attention[0],
      5 encoder_tokens = encoder_text,

1 frames
/usr/local/lib/python3.9/dist-packages/bertviz/model_view.py in model_view(attention, tokens, sentence_b_start, prettify_tokens, display_mode, encoder_attention, decoder_attention, cross_attention, encoder_tokens, decoder_tokens, include_layers, include_heads, html_action)
    128 if include_heads is None:
    129 include_heads = list(range(n_heads))
--> 130 encoder_attention = format_attention(encoder_attention, include_layers, include_heads)
    131 attn_data.append(
    132 {

/usr/local/lib/python3.9/dist-packages/bertviz/util.py in format_attention(attention, layers, heads)
      9 # 1 x num_heads x seq_len x seq_len
     10 if len(layer_attention.shape) != 4:
---> 11 raise ValueError("The attention tensor does not have the correct number of dimensions. Make sure you set "
     12 "output_attentions=True when initializing your model.")
     13 layer_attention = layer_attention.squeeze(0)

ValueError: The attention tensor does not have the correct number of dimensions. Make sure you set output_attentions=True when initializing your model.

```

---

<div class="post-metadata">

**Author:** ![dblakely](https://avatars.discourse-cdn.com/v4/letter/d/54ee81/32.png) [@dblakely](https://discuss.huggingface.co/u/dblakely)\
**Post date:** [April 5, 2023, 7:06pm UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/2 "2023-04-05T19:06:37Z")

</div>

Hi, what’s going on here is that when you get back the attentions from a `generate` call, you’re getting back a tuple of attentions for each token you generated, so that’s what the outer tuple is for. Meanwhile, the inner tuples are for each layer of the model.

For example if you run:

```python
from transformers import T5ForConditionalGeneration, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("t5-base")
model = T5ForConditionalGeneration.from_pretrained("t5-base")

inputs = tokenizer(
    "Translate from English to French: Hello, how are you?",
    return_tensors="pt",
)

output = model.generate(**inputs, return_dict_in_generate=True, output_attentions=True)

sequences = output.sequences

encoder_attentions = output.encoder_attentions
cross_attentions = output.cross_attentions
decoder_attentions = output.decoder_attentions

print(f"{sequences.shape = }")
print(f"{len(cross_attentions) = }")
print(f"{len(decoder_attentions) = }")

```

you’ll get:

```auto
sequences.shape = torch.Size([1, 10])
len(cross_attentions) = 9
len(decoder_attentions) = 9

```

Note that it’s actually `num_tokens_generated - 1` because the final token (usually the EOS token) is never fed through the decoder after it’s predicted, so there are no attention scores for it. So what you see is that len(cross\_attentions) = len(decoder\_attentions) = num tokens generated - 1.

Then if you check how long the inner tuples are, you get:

```python
len(cross_attentions[0]) = 12
len(decoder_attentions[0]) = 12

```

because there are 12 attention layers in `t5-base`.

In contrast, for the _encoder attentions_, you don’t get a nested tuple because the encoder is only used once at the beginning. You simply get a tuple containing the encoder attentions from each encoder layer (12 layers for `t5-base`).

Getting to the issue of how to make this work with Bertviz: the error you got from Bertviz says the decoder\_attentions need to be a tuple of tensors (1 tensor of attentions for each layer) and each tensor needs to have shape `(1, num_heads, decoder_sequence_length, decoder_sequence_length)`. In the example above, that would be `(1, 12, 9, 9)` (batch size 1, 12 heads, 9 positions attending to 9 positions). The tricky part is that the decoder attentions you get back from a `generate` call don’t have a consistent shape because the size of decoder sequence increases by 1 each time a new token is generated.

For example, you can check this like this:

```auto
decoder_attentions[0][0].shape = torch.Size([1, 12, 1, 1]) # token 0, layer 0
decoder_attentions[1][0].shape = torch.Size([1, 12, 1, 2]) # token 1, layer 0
decoder_attentions[2][0].shape = torch.Size([1, 12, 1, 3]) # token 2, layer 0

```

At each step, there’s one query token attending to the sequence that’s been written so far. But to work with Bertviz, the per-token attentions for each layer need to be combined into one tensor (by concatenating), but that’s tricky because you’d have to concatenate on dimension 2, but the tensors vary in shape along position 3, which will cause an pytorch error. It’d be possible to get around this by padding all the decoder attentions to the same length on dimension 3 (9 in this example) by using dummy a tensor of 0s. I.e., pad it so that the shape of `decoder_attentions[0][0]` goes from `[1, 12, 1, 1]` to `[1, 12, 1, 9]`.

Though ultimately it would probably be easiest to just pass your generated outputs through the model to get back attentions in a form that’s easier to work with for your purposes. For example, you could do something like this:

```python
# Generate
output = model.generate(**inputs, return_dict_in_generate=True)

# Pass the outputs through the model again to get the attentions
out = model(**inputs, decoder_input_ids=output.sequences, output_attentions=True, return_dict=True)

encoder_attentions = out.encoder_attentions
cross_attentions = out.cross_attentions
decoder_attentions = out.decoder_attentions

print(f"{len(encoder_attentions) = }")
print(f"{len(cross_attentions) = }")
print(f"{len(decoder_attentions) = }")

```

Then you get back:

```auto
len(encoder_attentions) = 12
len(cross_attentions) = 12
len(decoder_attentions) = 12

```

So type of attention has 12 tensors–1 for each layer–as requested by Bertviz and you no longer have the nested tuples from before. So feeding these outputs should work with Bertviz.

---

<div class="post-metadata">

**Author:** ![snork-maiden](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@snork-maiden](https://discuss.huggingface.co/u/snork-maiden)\
**Post date:** [April 6, 2023, 4:23pm UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/3 "2023-04-06T16:23:45Z")

</div>

Hello! Thank you soo much for your time and clear explanation! It has finally worked!

---

<div class="post-metadata">

**Author:** ![rr-7](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@rr-7](https://discuss.huggingface.co/u/rr-7)\
**Post date:** [March 17, 2024, 6:45am UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/4 "2024-03-17T06:45:13Z")

</div>

Hi @snork-maiden , Could you share your notebook if possible,  
I tried this but still getting error regarding mismatch in size in attention and number of tokens (I am using text to sql t5 model)  
ValueError: Attention has 17 positions, while number of tokens is 16 for tokens:

---

<div class="post-metadata">

**Author:** ![snork-maiden](https://avatars.discourse-cdn.com/v4/letter/s/c77e96/32.png) [@snork-maiden](https://discuss.huggingface.co/u/snork-maiden)\
**Post date:** [March 20, 2024, 8:28am UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/5 "2024-03-20T08:28:30Z")

</div>

Hello @rr-7 here is [part of my notebook](https://colab.research.google.com/drive/1qyKvMRTVYzoNXTMzis8NAF0JUc8UcdIF?usp=sharing) where I do visualization.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![tahmid1234](https://avatars.discourse-cdn.com/v4/letter/t/258eb7/32.png) [@tahmid1234](https://discuss.huggingface.co/u/tahmid1234)\
**Post date:** [October 22, 2025, 8:37am UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/6 "2025-10-22T08:37:25Z")

</div>

Hi @snork-maiden , thanks for sharing the notebook. How long did it take for the view to render? It’s been a while, and nothing is being displayed. The cell is still running and a dropdown for different attention layers has appeared.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 22, 2025, 9:13am UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/7 "2025-10-22T09:13:23Z")

</div>

I was able to download it normally, so [I tried re-uploading it](https://huggingface.co/datasets/John6666/forum2/blob/main/vis.ipynb).

---

<div class="post-metadata">

**Author:** ![tahmid1234](https://avatars.discourse-cdn.com/v4/letter/t/258eb7/32.png) [@tahmid1234](https://discuss.huggingface.co/u/tahmid1234)\
**Post date:** [October 22, 2025, 3:09pm UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/8 "2025-10-22T15:09:44Z")

</div>

Sorry, I’m not sure what you meant. My script seems to be working, but I just can’t see the visualization. The cell keeps running indefinitely, and when I tried to download the HTML file, it was 1.3 GB and couldn’t be opened.

My code -

`output = model.generate(**inputs`,  
`eos_token_id=tokenizer.eos_token_id,`  
`pad_token_id=tokenizer.pad_token_id,`  
`max_new_tokens= expected_lenght,`  
`return_dict_in_generate=True,`  
`num_beams=7,`  
`length_penalty=0.9,`  
`no_repeat_ngram_size=3,`  
`forced_eos_token_id=tokenizer.eos_token_id,)`

`out = model(**inputs, decoder_input_ids=output.sequences, output_attentions=True, return_dict=True)`

`encoder_attentions = out.encoder_attentions`  
`cross_attentions = out.cross_attentions`  
`decoder_attentions = out.decoder_attentions`

`print(f"{len(encoder_attentions) = }")`  
`print(f"{len(cross_attentions) = }")`  
`print(f"{len(decoder_attentions) = }")`

`i = inputs['input_ids'][0]`  
`encoder_text = tokenizer.convert_ids_to_tokens(i.tolist())`  
`decoder_text =tokenizer.convert_ids_to_tokens(output.sequences[0].tolist())`  
`view = model_view(`  
`encoder_attention=out.encoder_attentions,`  
`decoder_attention=out.decoder_attentions,`  
`cross_attention=out.cross_attentions,`  
`encoder_tokens= encoder_text,`  
`decoder_tokens = decoder_text,`  
`html_action='return'`  
`)`  
`display(HTML(view.data))`

`with open("t5_attention.html", "w", encoding="utf-8") as f:`  
` f.write(view.data)`

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 22, 2025, 3:17pm UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/9 "2025-10-22T15:17:22Z")

</div>

Oh… Sorry. I misread your post. You were talking about the time it takes to render, not the notebook’s loading time…

I’ll try it too.

Edit:  
[I supplemented the missing parts of the original notebook and ran it on Colab CPU](https://huggingface.co/datasets/John6666/forum2/blob/main/vis_fixed.ipynb). I think this should probably reproduce the rendering now…

---

<div class="post-metadata">

**Author:** ![tahmid1234](https://avatars.discourse-cdn.com/v4/letter/t/258eb7/32.png) [@tahmid1234](https://discuss.huggingface.co/u/tahmid1234)\
**Post date:** [October 23, 2025, 8:28pm UTC](https://discuss.huggingface.co/t/error-when-trying-to-visualize-attention-in-t5-model/35350/11 "2025-10-23T20:28:11Z")

</div>

Thanks it worked