# How to use custom positional embedding while fine tuning Bert

**URL:** <https://discuss.huggingface.co/t/how-to-use-custom-positional-embedding-while-fine-tuning-bert/7992>\
**Category:** Beginners\
**Created:** [July 4, 2021, 9:54am UTC](https://discuss.huggingface.co/t/how-to-use-custom-positional-embedding-while-fine-tuning-bert/7992 "2021-07-04T09:54:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![bengul](https://avatars.discourse-cdn.com/v4/letter/b/838e76/32.png) [@bengul](https://discuss.huggingface.co/u/bengul)\
**Post date:** [July 4, 2021, 9:54am UTC](https://discuss.huggingface.co/t/how-to-use-custom-positional-embedding-while-fine-tuning-bert/7992/1 "2021-07-04T09:54:42Z")

</div>

Hi, I want to fine-tune Bert for question answering but instead of using the built-in position embeddings (which are the indices of the tokens), I want to add my own positional embedding (for example the sinusoidal positional embedding mentioned in the paper “Attention is all you need”). I understand one way to do that would be to modify the input embeddings in the BertModel class, but I am not sure how to modify that class. If there are better ways, I would be happy to follow. Any suggestions would be highly appreciated.

---

<div class="post-metadata">

**Author:** ![JinKing](https://avatars.discourse-cdn.com/v4/letter/j/b5ac83/32.png) [@JinKing](https://discuss.huggingface.co/u/JinKing)\
**Post date:** [August 19, 2022, 7:42am UTC](https://discuss.huggingface.co/t/how-to-use-custom-positional-embedding-while-fine-tuning-bert/7992/2 "2022-08-19T07:42:39Z")

</div>

Did you solve this problem? I’m trying to solve same thing.

---

<div class="post-metadata">

**Author:** ![TimShieh](https://avatars.discourse-cdn.com/v4/letter/t/7feea3/32.png) [@TimShieh](https://discuss.huggingface.co/u/TimShieh)\
**Post date:** [September 14, 2022, 12:53pm UTC](https://discuss.huggingface.co/t/how-to-use-custom-positional-embedding-while-fine-tuning-bert/7992/3 "2022-09-14T12:53:07Z")

</div>

All you need is change BertEmbeddings class, you can build a new class and inherit BertEmbeddings.  
You can follow the post.(I wrote the solution in it.)

> [@How to use additional input features for NER?](https://discuss.huggingface.co/t/how-to-use-additional-input-features-for-ner/4364/26):
>
> Hi f3n1Xx, I’m still stuck at the same problem and it’s also for a biological problem. There is no code in the link you provided. Would you mind posting here how you added additional features. If you can’t release the entire code could you just show a simplified example please? Thank you!