# About the Cross-attention Layer Shape in Encoder-Decoder Model

**URL:** <https://discuss.huggingface.co/t/about-the-cross-attention-layer-shape-in-encoder-decoder-model/15814>\
**Category:** 🤗Transformers\
**Created:** [March 17, 2022, 4:45pm UTC](https://discuss.huggingface.co/t/about-the-cross-attention-layer-shape-in-encoder-decoder-model/15814 "2022-03-17T16:45:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![zhkleciel](https://avatars.discourse-cdn.com/v4/letter/z/4af34b/32.png) [@zhkleciel](https://discuss.huggingface.co/u/zhkleciel)\
**Post date:** [March 17, 2022, 4:45pm UTC](https://discuss.huggingface.co/t/about-the-cross-attention-layer-shape-in-encoder-decoder-model/15814/1 "2022-03-17T16:45:15Z")

</div>

I am trying to intialize a bert2bert model with **bert-base-uncased** as encoder and **bert-large-uncased** as decoder with the following codes:

```auto
from transformers import EncoderDecoderModel, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = EncoderDecoderModel.from_encoder_decoder_pretrained(
    "bert-base-uncased", "bert-large-uncased") 

```

When I print the all layers of the model, the results are:

```auto
...
(attention): BertAttention(
  (self): BertSelfAttention(
    (query): Linear(in_features=1024, out_features=1024, bias=True)
    (key): Linear(in_features=1024, out_features=1024, bias=True)
    (value): Linear(in_features=1024, out_features=1024, bias=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
  (output): BertSelfOutput(
    (dense): Linear(in_features=1024, out_features=1024, bias=True)
    (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
)
(crossattention): BertAttention(
  (self): BertSelfAttention(
    (query): Linear(in_features=1024, out_features=1024, bias=True)
    (key): Linear(in_features=1024, out_features=1024, bias=True)
    (value): Linear(in_features=1024, out_features=1024, bias=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
  (output): BertSelfOutput(
    (dense): Linear(in_features=1024, out_features=1024, bias=True)
    (LayerNorm): LayerNorm((1024,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
)
...

```

For the cross attention part, I think the shape for query and key should be (768,1024) instead of (1024,1024), since it converts the bert-base output dim (768) to bert-large dim (1024). Who can tell me where I made a mistake and give me a explanation? Many thanks!

---

<div class="post-metadata">

**Author:** ![zhkleciel](https://avatars.discourse-cdn.com/v4/letter/z/4af34b/32.png) [@zhkleciel](https://discuss.huggingface.co/u/zhkleciel)\
**Post date:** [March 18, 2022, 2:25am UTC](https://discuss.huggingface.co/t/about-the-cross-attention-layer-shape-in-encoder-decoder-model/15814/2 "2022-03-18T02:25:00Z")

</div>

I found the codes in EncoderDecoderModel class that maps the encoder hidden state size to decoder hidden state size in the link [here](https://github.com/huggingface/transformers/blob/198c335d219a5eb4d3f124fdd1ce1a9cd9f78a9b/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L506). The problem solved.