# Add per-word embedding from outer source to Bert embedding layer

**URL:** <https://discuss.huggingface.co/t/add-per-word-embedding-from-outer-source-to-bert-embedding-layer/69064>\
**Category:** Beginners\
**Created:** [January 14, 2024, 9:00am UTC](https://discuss.huggingface.co/t/add-per-word-embedding-from-outer-source-to-bert-embedding-layer/69064 "2024-01-14T09:00:33Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![orko](https://avatars.discourse-cdn.com/v4/letter/o/b5a626/32.png) [@orko](https://discuss.huggingface.co/u/orko)\
**Post date:** [January 14, 2024, 9:00am UTC](https://discuss.huggingface.co/t/add-per-word-embedding-from-outer-source-to-bert-embedding-layer/69064/1 "2024-01-14T09:00:34Z")

</div>

Hi  
I have an outer source with per-word embedding,  
I want to add it as an additional later for the bert embddings:  
Such that  
custom\_embedding\_layer = (word\_embeddings)+(position\_embeddings)+(outersource\_embeddings)

I saw the answer in:

> [@How to use additional input features for NER?](https://discuss.huggingface.co/t/how-to-use-additional-input-features-for-ner/4364/2):
>
> Actually no, because the pre-trained tokenizer only knows tokens, not tokens + POS tags. A better way to do this would be to create an additional input to the model (besides input\_ids and token\_type\_ids) called pos\_tag\_ids, for which you can add an additional embedding layer (nn.Embedding). In that way, you can sum the embeddings of the tokens, token types and the POS tags. Let’s illustrate this for a pre-trained BERT model: We first have to modify the [BertEmbeddings class](https://github.com/huggingface/transformers/blob/4c32f9f26e6a84f0d9843fec8757e6ce640bb44e/src/transformers/models/bert/modeling_bert.py#L166). In short, we’ll add…

But I would rather create a wrapper class rather then change the source code of BertEmbeddings itself  
How can I do it?  
Thanks