# Custom BERT Masked LM output embedding

**URL:** <https://discuss.huggingface.co/t/custom-bert-masked-lm-output-embedding/22656>\
**Category:** Beginners\
**Created:** [September 7, 2022, 3:20am UTC](https://discuss.huggingface.co/t/custom-bert-masked-lm-output-embedding/22656 "2022-09-07T03:20:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![kevintuan](https://avatars.discourse-cdn.com/v4/letter/k/3bc359/32.png) [@kevintuan](https://discuss.huggingface.co/u/kevintuan)\
**Post date:** [September 7, 2022, 3:20am UTC](https://discuss.huggingface.co/t/custom-bert-masked-lm-output-embedding/22656/1 "2022-09-07T03:20:49Z")

</div>

Hi,  
I trained a custom sense embeddings based on Wordnet definition and tree structure.  
Now I want to test the embeddings by fine tuning BERT masked LM so the model predicts the most likely sense embedding. For example:  
" I need to go to the [bank] today" → bank.wn.02

I’m uncertain how to accomplish this.

1. Can I provide a set of output labels with their embeddings different from the input embedding and input\_ids?
2. Should I build a custom softmax layer from the output\_hidden\_states of the BERTModel