# Training BERT for word embedding **URL:** \ **Category:** Beginners\ **Created:** [July 13, 2022, 9:39am UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322 "2022-07-13T09:39:52Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![joval](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joval/32/5335_2.png) [@joval](https://discuss.huggingface.co/u/joval)\ **Post date:** [July 13, 2022, 9:39am UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/1 "2022-07-13T09:39:52Z")
Hello everyone, Please I’m not familiar with BERT, but I’ll like to train a BERT model just for word embedding (not NSP or MLM), in order to compare its impact on some task (I can give details if needed) against W2V. In my case, I’ll like to train BERT on my dataset, but what I can find in the research is _how to train BERT for MLM for example_. So I don’t how to use this model to embed word. Can someone help me please, to archive this goal? ---
**Author:** ![Nlpeva](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nlpeva/32/5309_2.png) [@Nlpeva](https://discuss.huggingface.co/u/Nlpeva)\ **Post date:** [July 13, 2022, 12:21pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/2 "2022-07-13T12:21:09Z")
Please feel free to share more details. 🙂 It really depends a lot on the context around the sentences/words you want to embed. Hopefully these two discussions can help: 🤗 > [@Generate raw word embeddings using transformer models like BERT for downstream process](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958): > > Hi, I am new to using transformer based models. I have a few basic questions, hopefully, someone can shed light, please. I’ve been training GloVe and word2vec on my corpus to generate word embedding, where a unique word has a vector to use in the downstream process. Now, my questions are: Can we generate a similar embedding using the BERT model on the same corpus? Can we have one unique word with its vector? BERT is contextual, not sure how the vector will look like for the same word which i… > [@Obtaining word-embeddings from Roberta](https://discuss.huggingface.co/t/obtaining-word-embeddings-from-roberta/7735): > > Hello Everyone, I am fine-tuning a pertained masked LM (distil-roberta) on a custom dataset. Post-training, I would like to use the word embeddings in a downstream task. How does one go about obtaining embeddings for whole-words when the model uses sub-word tokenising. For example, tokeniser.tokenize(‘floral’) will give me [‘fl’, ‘oral’]. So, if ‘floral’ is not even a part of the vocabulary, how do I obtain its embedding? When I do this: tokens = tokenizer.encode("floral") word = tokenizer.e… ---
**Author:** ![joval](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joval/32/5335_2.png) [@joval](https://discuss.huggingface.co/u/joval)\ **Post date:** [July 13, 2022, 12:48pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/3 "2022-07-13T12:48:37Z")
Hello @Nlpeva and thanks for your response. In my case, I’d like to use BERT and W2V in Word Sense Disambiguation for ambiguous queries (Information Retrieval). The goal with these models is to embed all context words of the ambiguous word. - So with W2V, I’ve just built a list of all sentences in my dataset and used gensim package to train a W2V model on these sentences. - In the case of BERT, I’ll like to do the same, just to pass a list of sentences and have at the end of training a model like the original BERT and embed different words By the way, I’d like to note I’ve been restricted by my supervisor to just use the original BERT model and to train new one on my dataset. Thank again for the first link, it’ll help me to embed word after training the new BERT on my dataset, but now I’ve to first train the model. ---
**Author:** ![Nlpeva](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nlpeva/32/5309_2.png) [@Nlpeva](https://discuss.huggingface.co/u/Nlpeva)\ **Post date:** [July 13, 2022, 11:18pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/4 "2022-07-13T23:18:01Z")
Hmm, have you looked at spacy-transformers? That might be a good fit for your project… Here’s also a paper I read. They tried fine-tuning BERT on the task of predicting the meaning of the ambiguous word. > **[GitHub - BPYap/BERT-WSD: \[EMNLP-Findings 2020\] Adapting BERT for Word Sense...](https://github.com/BPYap/BERT-WSD)** > > \[EMNLP-Findings 2020\] Adapting BERT for Word Sense Disambiguation with Gloss Selection Objective and Example Sentences - GitHub - BPYap/BERT-WSD: \[EMNLP-Findings 2020\] Adapting BERT for Word Sense ... That might be a bit too in-depth for what your supervisor wanted, though! 🤗 ---
**Author:** ![cog](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@cog](https://discuss.huggingface.co/u/cog)\ **Post date:** [July 14, 2022, 2:24am UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/5 "2022-07-14T02:24:20Z")
hi @joval There HF docs show [BertForMaskedLM](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForMaskedLM) parameter and output. you can train BERT MLM from scratch with that class. Thanks for _nielsr_, there some good tutorial of fine tuning BERT with HF. It will be help to you underestand whole train structure. > **[GitHub - NielsRogge/Transformers-Tutorials: This repository contains demos I...](https://github.com/NielsRogge/Transformers-Tutorials)** > > This repository contains demos I made with the Transformers library by HuggingFace. - GitHub - NielsRogge/Transformers-Tutorials: This repository contains demos I made with the Transformers library... regards. ---
**Author:** ![joval](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joval/32/5335_2.png) [@joval](https://discuss.huggingface.co/u/joval)\ **Post date:** [July 14, 2022, 7:43am UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/6 "2022-07-14T07:43:26Z")
Hi @cog . Thank you for your response. But as explained in my first message, I’m not looking for BertForMaskedLM, because I don’t aim to use it. I’ld like to use BERT just to embed words, not to predict masked words. ---
**Author:** ![joval](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joval/32/5335_2.png) [@joval](https://discuss.huggingface.co/u/joval)\ **Post date:** [July 14, 2022, 8:13am UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/7 "2022-07-14T08:13:43Z")
Thank you @Nlpeva for all these resources. The only problem I’ve with all this is that, For my WSD in IR, I already have an existing unsupervised process (configured with W2V), the goal is just to see the impact of other models (especially BERT, as it is supposed to produce best result than W2V) of word embedding. And to clarify, The process (with W2V) has already been validated with the produced results. So to use BERT I can only adapt it to that process, by embedding context words. ---
**Author:** ![Nlpeva](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nlpeva/32/5309_2.png) [@Nlpeva](https://discuss.huggingface.co/u/Nlpeva)\ **Post date:** [July 14, 2022, 5:10pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/8 "2022-07-14T17:10:17Z")
What do you think about using Sentence-BERT? [https://www.sbert.net/examples/applications/computing-embeddings/README.html#sentence-embeddings-with-transformers](https://www.sbert.net/examples/applications/computing-embeddings/README.html#sentence-embeddings-with-transformers) ---
**Author:** ![joval](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joval/32/5335_2.png) [@joval](https://discuss.huggingface.co/u/joval)\ **Post date:** [July 14, 2022, 5:36pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/9 "2022-07-14T17:36:21Z")
Hi @Nlpeva , I think it can be useful for WSD. But, as I described, I don’t want to embed sentences but words. So using it will force me to change my approach, which I can’t because the idea here is to compare the impact of bert and W2V on this approach ---
**Author:** ![pritamdeka](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pritamdeka/32/5418_2.png) [@pritamdeka](https://discuss.huggingface.co/u/pritamdeka)\ **Post date:** [July 19, 2022, 12:05pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/10 "2022-07-19T12:05:24Z")
Hi. Sentence BERT is useful for words as well as sentences. You can use it to get word embeddings instead of sentence embeddings. ---
**Author:** ![joval](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joval/32/5335_2.png) [@joval](https://discuss.huggingface.co/u/joval)\ **Post date:** [July 19, 2022, 12:35pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/11 "2022-07-19T12:35:18Z")
Hi @pritamdeka , Yes, that’s true and thank you. But what I’d like is **how to train BERT on a new dataset** , not to use the already pre-train BERT. And let me notice that, I have data annotated neither for NSP nor for MLM. It why I’m asking if it’s possible and if yes, how to do it? ---
**Author:** ![pritamdeka](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pritamdeka/32/5418_2.png) [@pritamdeka](https://discuss.huggingface.co/u/pritamdeka)\ **Post date:** [July 19, 2022, 1:37pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/12 "2022-07-19T13:37:03Z")
Would it be possible to know how the annotated data looks like? ---
**Author:** ![joval](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joval/32/5335_2.png) [@joval](https://discuss.huggingface.co/u/joval)\ **Post date:** [July 19, 2022, 1:49pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/13 "2022-07-19T13:49:23Z")
For which task? ---
**Author:** ![sdegrace](https://avatars.discourse-cdn.com/v4/letter/s/f475e1/32.png) [@sdegrace](https://discuss.huggingface.co/u/sdegrace)\ **Post date:** [July 21, 2022, 5:57pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/14 "2022-07-21T17:57:23Z")
I think there might be a bit of confusion about what BERT is. BERT was trained using MLM and next sentence prediction. You can fine-tune using MLM alone for simplicity’s sake. Once you have finished fine-tuning, all you have to do is grab the embeddings from the model before it’s passed into the MLM head. You can do this by specifying `output_hidden_states=True` when calling the model. Read more in the docs! > **[BERT](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertForMaskedLM.forward)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![joval](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joval/32/5335_2.png) [@joval](https://discuss.huggingface.co/u/joval)\ **Post date:** [July 25, 2022, 1:54pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/15 "2022-07-25T13:54:27Z")
Hello @sdegrace Thank you for your reply. Let me explore this. ---
**Author:** ![dinipuspita](https://avatars.discourse-cdn.com/v4/letter/d/b2d939/32.png) [@dinipuspita](https://discuss.huggingface.co/u/dinipuspita)\ **Post date:** [November 7, 2022, 10:50am UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/16 "2022-11-07T10:50:50Z")
Hello @joval I am very interest in your topic about training BERT for word embedding. If you don’t mind, I want to know the updates about this. Can BERT be used just for word embedding? Because my thesis related to this and this is my first time using BERT. The research that I find is how to train it only. I still haven’t found out how to train BERT for word embedding only. Thanks in advance! 😃 ---
**Author:** ![joval](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joval/32/5335_2.png) [@joval](https://discuss.huggingface.co/u/joval)\ **Post date:** [November 9, 2022, 6:26pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/17 "2022-11-09T18:26:00Z")
Hi @dinipuspita. To archieve this, I trained BERT using MLM ([the link used](https://www.thepythoncode.com/article/pretraining-bert-huggingface-transformers-in-python)) and and I applied @sdegrace’s suggestion : > Once you have finished fine-tuning, all you have to do is grab the embeddings from the model before it’s passed into the MLM head. You can do this by specifying `output_hidden_states=True` when calling the model. ---
**Author:** ![dinipuspita](https://avatars.discourse-cdn.com/v4/letter/d/b2d939/32.png) [@dinipuspita](https://discuss.huggingface.co/u/dinipuspita)\ **Post date:** [November 12, 2022, 12:55pm UTC](https://discuss.huggingface.co/t/training-bert-for-word-embedding/20322/18 "2022-11-12T12:55:18Z")
Hi @joval thanks for the reply. I have followed question. After I add `output_hidden_states=True` parameter to my model, how can I see the result of embedding? Thank you.