# \`BertEmbeddings\` contains positional embedding?

**URL:** <https://discuss.huggingface.co/t/bertembeddings-contains-positional-embedding/28442>\
**Category:** 🤗Transformers\
**Created:** [December 26, 2022, 10:36am UTC](https://discuss.huggingface.co/t/bertembeddings-contains-positional-embedding/28442 "2022-12-26T10:36:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ptej](https://avatars.discourse-cdn.com/v4/letter/p/dec6dc/32.png) [@ptej](https://discuss.huggingface.co/u/ptej)\
**Post date:** [December 26, 2022, 10:36am UTC](https://discuss.huggingface.co/t/bertembeddings-contains-positional-embedding/28442/1 "2022-12-26T10:36:15Z")

</div>

Hello,

I am using `BertEmbeddings` from `transformers.models.bert.modeling_bert` for a custom model.

I wasn’t sure if the output includes the sin/cos positional encoding ([Attention Is All You Need](https://arxiv.org/abs/1706.03762)), or if I need to add the position encoding myself.

```python
from transformers import BertConfig
from transformers.models.bert.modeling_bert import BertEmbeddings

bert_config = BertConfig("bert-base-uncased")
bert_embeds = BertEmbeddings(bert_config)

text_embeddings = bert_embeds(text_ids)

text_embeddings # Does this include positional embeddings?
# Or should I do
final_embeddings = text_embeddings + positional_embeddings

```

Thanks!  
Priyam

---

<div class="post-metadata">

**Author:** ![NimaBoscarino](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimaboscarino/32/4358_2.png) [@NimaBoscarino](https://discuss.huggingface.co/u/NimaBoscarino)\
**Post date:** [December 26, 2022, 7:19pm UTC](https://discuss.huggingface.co/t/bertembeddings-contains-positional-embedding/28442/2 "2022-12-26T19:19:09Z")

</div>

Hello there! `BertEmbeddings` has the positional embeddings for you: [transformers/modeling\_bert.py at 31d452c68b34c2567b62924ee0df40a83cbc52d5 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/31d452c68b34c2567b62924ee0df40a83cbc52d5/src/transformers/models/bert/modeling_bert.py#L187)

We use a vanilla nn.Embedding layer instead of the sin/cos positional encoding (more on that here: [Why positional embeddings are implemented as just simple embeddings? - #6 by yjernite](https://discuss.huggingface.co/t/why-positional-embeddings-are-implemented-as-just-simple-embeddings/585/6)). If you want to override that and use sinusoidal embeddings though, you can follow the tip here: [How to use custom positional embedding while fine tuning Bert - #2 by JinKing](https://discuss.huggingface.co/t/how-to-use-custom-positional-embedding-while-fine-tuning-bert/7992/2)

Hope that helps! Cheers!

---

<div class="post-metadata">

**Author:** ![ptej](https://avatars.discourse-cdn.com/v4/letter/p/dec6dc/32.png) [@ptej](https://discuss.huggingface.co/u/ptej)\
**Post date:** [December 27, 2022, 7:03am UTC](https://discuss.huggingface.co/t/bertembeddings-contains-positional-embedding/28442/3 "2022-12-27T07:03:43Z")

</div>

Extremely helpful. Thanks!