# BertSelfAttention, BertSelfOutput implementation

**URL:** <https://discuss.huggingface.co/t/bertselfattention-bertselfoutput-implementation/21439>\
**Category:** 🤗Transformers\
**Created:** [August 9, 2022, 7:49pm UTC](https://discuss.huggingface.co/t/bertselfattention-bertselfoutput-implementation/21439 "2022-08-09T19:49:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![heekang](https://avatars.discourse-cdn.com/v4/letter/h/51bf81/32.png) [@heekang](https://discuss.huggingface.co/u/heekang)\
**Post date:** [August 9, 2022, 7:49pm UTC](https://discuss.huggingface.co/t/bertselfattention-bertselfoutput-implementation/21439/1 "2022-08-09T19:49:19Z")

</div>

Hi, I’m studying BERT and am curious about the implementation of `BertSelfAttention` and `BertSelfOutput`.

According to the Transformer paper([Attention Is All You Need (2017, Vaswani et el.)](https://arxiv.org/abs/1706.03762)), multi-head self attention requires 4 Linear layers: one for query, one for key, one for value, and the last for final output.

However, `BertSelfAttention` only has three Linear layers; only for query, key, and value. The Linear layer for the final output is in `BertSelfOutput`.

Is there any reason why they are implemented like this?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [August 10, 2022, 9:51am UTC](https://discuss.huggingface.co/t/bertselfattention-bertselfoutput-implementation/21439/2 "2022-08-10T09:51:41Z")

</div>

I think this was a design decision by @thomwolf.

---

<div class="post-metadata">

**Author:** ![thomwolf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomwolf/32/2198_2.png) [@thomwolf](https://discuss.huggingface.co/u/thomwolf)\
**Post date:** [August 10, 2022, 12:33pm UTC](https://discuss.huggingface.co/t/bertselfattention-bertselfoutput-implementation/21439/3 "2022-08-10T12:33:17Z")

</div>

It was done like this in the original tensorflow implementation of Bert and can probably be traced back to the tensor2tensor library

---

<div class="post-metadata">

**Author:** ![ccdv](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ccdv/32/28048_2.png) [@ccdv](https://discuss.huggingface.co/u/ccdv)\
**Post date:** [August 10, 2022, 4:33pm UTC](https://discuss.huggingface.co/t/bertselfattention-bertselfoutput-implementation/21439/4 "2022-08-10T16:33:03Z")

</div>

Its model/original implementation dependent, DistilBERT does the 4th linear transformation for example.

---

<div class="post-metadata">

**Author:** ![heekang](https://avatars.discourse-cdn.com/v4/letter/h/51bf81/32.png) [@heekang](https://discuss.huggingface.co/u/heekang)\
**Post date:** [August 11, 2022, 7:31am UTC](https://discuss.huggingface.co/t/bertselfattention-bertselfoutput-implementation/21439/5 "2022-08-11T07:31:58Z")

</div>

Thank you all for your reply:-)