# Sizes of Query, key and value vector in Bert Model

**URL:** <https://discuss.huggingface.co/t/sizes-of-query-key-and-value-vector-in-bert-model/1102>\
**Category:** 🤗Transformers\
**Created:** [September 13, 2020, 4:10pm UTC](https://discuss.huggingface.co/t/sizes-of-query-key-and-value-vector-in-bert-model/1102 "2020-09-13T16:10:13Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Khondoker](https://avatars.discourse-cdn.com/v4/letter/k/82dd89/32.png) [@Khondoker](https://discuss.huggingface.co/u/Khondoker)\
**Post date:** [September 13, 2020, 4:10pm UTC](https://discuss.huggingface.co/t/sizes-of-query-key-and-value-vector-in-bert-model/1102/1 "2020-09-13T16:10:13Z")

</div>

I have a question about the sizes of query, key and value vectors. As mentioned in this [paper](https://arxiv.org/pdf/1906.04341.pdf) and also demonstrated in this [medium](https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1), we should be expecting the sizes of query, key and value vectors as [seq\_length x seq\_length]. But when I print the sizes of the parameter like below, I see the sizes of those vectors as [768 x 768].

```
for name, param in model.named_parameters():                

    print(name, param.size())

>>> bert.bert.encoder.layer.0.attention.self.query.weight torch.Size([768, 768])
       bert.bert.encoder.layer.0.attention.self.key.weight torch.Size([768, 768])
       bert.bert.encoder.layer.0.attention.self.value.weight torch.Size([768, 768])

```

I am really confused. I feel like I am missing something, could someone please help me figure it out?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [September 14, 2020, 1:41pm UTC](https://discuss.huggingface.co/t/sizes-of-query-key-and-value-vector-in-bert-model/1102/2 "2020-09-14T13:41:34Z")

</div>

You are looking at the weights of the query/key/value heads, not the value vectors.

---

<div class="post-metadata">

**Author:** ![Khondoker](https://avatars.discourse-cdn.com/v4/letter/k/82dd89/32.png) [@Khondoker](https://discuss.huggingface.co/u/Khondoker)\
**Post date:** [September 14, 2020, 3:42pm UTC](https://discuss.huggingface.co/t/sizes-of-query-key-and-value-vector-in-bert-model/1102/3 "2020-09-14T15:42:07Z")

</div>

Oh I see. Could you please mind explaining how vectors are derived from this weights? or give any references for it?

Thank you

---

<div class="post-metadata">

**Author:** ![simonschoe](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/simonschoe/32/20951_2.png) [@simonschoe](https://discuss.huggingface.co/u/simonschoe)\
**Post date:** [March 25, 2021, 9:43am UTC](https://discuss.huggingface.co/t/sizes-of-query-key-and-value-vector-in-bert-model/1102/4 "2021-03-25T09:43:29Z")

</div>

If you dive into the source code of `class BertSelfAttention(nn.Module)` you will find:

```
    self.num_attention_heads = config.num_attention_heads
    self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
    self.all_head_size = self.num_attention_heads * self.attention_head_size

    self.query = nn.Linear(config.hidden_size, self.all_head_size)
    self.key = nn.Linear(config.hidden_size, self.all_head_size)
    self.value = nn.Linear(config.hidden_size, self.all_head_size)

```

In my view, this illustrates that the size of Q, K and V in a single head is actually `768 x 64` if the number of heads is 12. Due to the multi-head attention these are later concatenated to again produce the abovementioned shape of `768 x 768` which you have obeserved.