# Bert embedding layer

**URL:** <https://discuss.huggingface.co/t/bert-embedding-layer/13355>\
**Category:** Beginners\
**Created:** [January 4, 2022, 8:46am UTC](https://discuss.huggingface.co/t/bert-embedding-layer/13355 "2022-01-04T08:46:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ShivaniSri](https://avatars.discourse-cdn.com/v4/letter/s/3ab097/32.png) [@ShivaniSri](https://discuss.huggingface.co/u/ShivaniSri)\
**Post date:** [January 4, 2022, 8:46am UTC](https://discuss.huggingface.co/t/bert-embedding-layer/13355/1 "2022-01-04T08:46:49Z")

</div>

I have taken specific word embeddings and considered bert model with those embeddings

self.bert = BertModel.from\_pretrained(‘bert-base-uncased’)  
self.bert(inputs\_embeds=x,attention\_mask=attention\_mask, \*args, \*\*kwargs)

Does this means I’m replacing the bert input embeddings(token+position+segment embeddings)

How to consider all embeddings i.e.,(token+position+segment+custom embeddings)

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [January 4, 2022, 4:54pm UTC](https://discuss.huggingface.co/t/bert-embedding-layer/13355/2 "2022-01-04T16:54:19Z")

</div>

Hi,

As you can see [here](https://github.com/huggingface/transformers/blob/19d37c2dd36d73537a6855c56808e524fb584459/src/transformers/models/bert/modeling_bert.py#L213), if you provide `inputs_embeds` yourself, they will only be used to replace the token embeddings. The token type and position embeddings will be added separately.