# Swapping out self-attention layer in BERT

**URL:** <https://discuss.huggingface.co/t/swapping-out-self-attention-layer-in-bert/29398>\
**Category:** Research\
**Created:** [January 11, 2023, 7:44pm UTC](https://discuss.huggingface.co/t/swapping-out-self-attention-layer-in-bert/29398 "2023-01-11T19:44:12Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sidkk86](https://avatars.discourse-cdn.com/v4/letter/s/b5e925/32.png) [@sidkk86](https://discuss.huggingface.co/u/sidkk86)\
**Post date:** [January 11, 2023, 7:44pm UTC](https://discuss.huggingface.co/t/swapping-out-self-attention-layer-in-bert/29398/1 "2023-01-11T19:44:12Z")

</div>

Hi team, I am looking to swap out the self attention layer in the BERT construction, and just retrain the embeddings with all other parts as is. I basically want to swap out [these](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L323-L352) 20 lines.

Is it possible for me to write my own self attention module, keep everything else the same and retrain the BERT embeddings? (I have high confidence that it is, but looking hopefully for instant gratification than sifting through 1000s of lines of code :D. Ideally, I think I would write my own module like [this](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L243) one and just wire it into the current pipeline ) Just scoping out the effort for this