# Google T5 cross\_attentions output

**URL:** <https://discuss.huggingface.co/t/google-t5-cross-attentions-output/104541>\
**Category:** Models\
**Created:** [August 29, 2024, 5:10pm UTC](https://discuss.huggingface.co/t/google-t5-cross-attentions-output/104541 "2024-08-29T17:10:36Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Yevhenii247](https://avatars.discourse-cdn.com/v4/letter/y/b5a626/32.png) [@Yevhenii247](https://discuss.huggingface.co/u/Yevhenii247)\
**Post date:** [August 29, 2024, 5:10pm UTC](https://discuss.huggingface.co/t/google-t5-cross-attentions-output/104541/1 "2024-08-29T17:10:36Z")

</div>

Hi everyone! I am trying to build attention heat map for google T5 small model ( for translation task ). But, I find the structure of cross\_attention quite misleading

```auto
cross_attentions = outputs.cross_attentions

```

Moreover, it’s not aligned with official documentation as well:

> cross\_attentions (tuple(torch.FloatTensor), optional, returned when output\_attentions=True is passed or when config.output\_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch\_size, num\_heads, sequence\_length, sequence\_length).

Attentions weights of the decoder’s cross-attention layer, after the attention softmax, used to compute the weighted average in the cross-attention heads.

After my investigation the actual output of cross\_attentions is tuple of len(num\_ouput\_tokens - 1). Each of **cross\_attentions[I]** consist of [num\_heads] (in case of small model - 6) tuples where each element is tensor of size (batch\_size, 8, 1, num\_of\_input\_tokens), where I find hard times to figure out what 8 actually is.

I would highly appreciate any explanation of how should I deal with this nested output to build heat maps similar to these:

 ![Screenshot 2024-08-29 at 20.07.09](https://us1.discourse-cdn.com/hellohellohello/original/3X/9/3/93963a363b2f9c53f962cd42733b4b775fdb3431.jpeg)