# Generate raw word embeddings using transformer models like BERT for downstream process

**URL:** <https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958>\
**Category:** Beginners\
**Created:** [December 25, 2020, 1:50am UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958 "2020-12-25T01:50:08Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![qandaleeb123](https://avatars.discourse-cdn.com/v4/letter/q/3e96dc/32.png) [@qandaleeb123](https://discuss.huggingface.co/u/qandaleeb123)\
**Post date:** [December 25, 2020, 1:50am UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/1 "2020-12-25T01:50:08Z")

</div>

Hi,  
I am new to using transformer based models. I have a few basic questions, hopefully, someone can shed light, please.

I’ve been training GloVe and word2vec on my corpus to generate word embedding, where a unique word has a vector to use in the downstream process. Now, my questions are:

1. Can we generate a similar embedding using the BERT model on the same corpus?
2. Can we have one unique word with its vector? BERT is contextual, not sure how the vector will look like for the same word which is repeated in different sentences.
3. If a word is repeated and not unique, not sure how I can use these vectors in the downstream process.

Appreciate your valuable inputs. I tried to look over the internet but was not able to find a clear answer. If someone can help with the above it will be really helpful.

Thanks

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [December 26, 2020, 8:54am UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/2 "2020-12-26T08:54:03Z")

</div>

1. Yes you can get a word embedding for a specific word in a sentence. You have to take care though, because in language models we often use a subword tokenizer. It chops words into smaller pieces. That means that you do not necessarily get one output for every word in a sentence, but probably more than one, namely one for all its subword components. What we then typically do is average the outputs of those tokens of the right word, to get one representation for that word. I’m on mobile now, but this is a modified script that I have used in the past to get the output of a specific word.

```python
 import numpy as np
 import torch
 from transformers import AutoTokenizer, AutoModel
 
 
 def get_word_idx(sent: str, word: str):
     return sent.split(" ").index(word)
 
 
 def get_hidden_states(encoded, token_ids_word, model, layers):
     """Push input IDs through model. Stack and sum `layers` (last four by default).
        Select only those subword token outputs that belong to our word of interest
        and average them."""
     with torch.no_grad():
         output = model(**encoded)
 
     # Get all hidden states
     states = output.hidden_states
     # Stack and sum all requested layers
     output = torch.stack([states[i] for i in layers]).sum(0).squeeze()
     # Only select the tokens that constitute the requested word
     word_tokens_output = output[token_ids_word]
 
     return word_tokens_output.mean(dim=0)
 
 
 def get_word_vector(sent, idx, tokenizer, model, layers):
     """Get a word vector by first tokenizing the input sentence, getting all token idxs
        that make up the word of interest, and then `get_hidden_states`."""
     encoded = tokenizer.encode_plus(sent, return_tensors="pt")
     # get all token idxs that belong to the word of interest
     token_ids_word = np.where(np.array(encoded.word_ids()) == idx)
 
     return get_hidden_states(encoded, token_ids_word, model, layers)
 
 
 def main(layers=None):
     # Use last four layers by default
     layers = [-4, -3, -2, -1] if layers is None else layers
     tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
     model = AutoModel.from_pretrained("bert-base-cased", output_hidden_states=True)
 
     sent = "I like cookies ." 
     idx = get_word_idx(sent, "cookies")

     word_embedding = get_word_vector(sent, idx, tokenizer, model, layers)
     
    return word_embedding 
 
 
 if __name__ == ' __main__':
     main()

```

1. Word embeddings are always contextual. You can extract values from the embedding layer only but that seems counter intuitive and will probably not work well. The whole point of (bidirectional) is to include context.
2. Not sure what you mean here. Unique in that sentence or unique in what sense?

---

<div class="post-metadata">

**Author:** ![qandaleeb123](https://avatars.discourse-cdn.com/v4/letter/q/3e96dc/32.png) [@qandaleeb123](https://discuss.huggingface.co/u/qandaleeb123)\
**Post date:** [December 26, 2020, 7:24pm UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/3 "2020-12-26T19:24:05Z")

</div>

Thanks! This is really helpful.

For unique word, i was trying to relate how GloVe has one word and its vector. The reason i was mentioning this is I wanted to combine GloVe and Transformer embeddings for a word in the sentence in the initial layer similar to this paper ([https://medinform.jmir.org/2019/4/e14850/](https://medinform.jmir.org/2019/4/e14850/)) and was not sure how to do this. I think somehow use transformer dynamically to get the word embedding and combine with Glove on the fly i guess.

---

<div class="post-metadata">

**Author:** ![ockapuh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ockapuh/32/33342_2.png) [@ockapuh](https://discuss.huggingface.co/u/ockapuh)\
**Post date:** [June 7, 2021, 8:52am UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/4 "2021-06-07T08:52:48Z")

</div>

Hi, Bram. Is it possible to generate this contextual embeddings for span of various tokens instead of for a single token? For example, in “three days ago I ate meat”, I would like to get contextual embeddings for “three days ago”. If possible, how should i do it?

Thank you so much.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [June 14, 2021, 8:35am UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/5 "2021-06-14T08:35:27Z")

</div>

Basically the same as above for each word in the span, and then sum/average those word vectors so you get one representation for the whole thing. Depending on your downstream task, this may not work very well, though.

---

<div class="post-metadata">

**Author:** ![Rainiefantasy](https://avatars.discourse-cdn.com/v4/letter/r/9dc877/32.png) [@Rainiefantasy](https://discuss.huggingface.co/u/Rainiefantasy)\
**Post date:** [September 5, 2021, 10:50pm UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/6 "2021-09-05T22:50:27Z")

</div>

Hi Bram,  
Just trying to use these BERT (vblagoje’s finetuned model) embedding outputs for downstream POS tagging ( using as input to logistic regression layer with their tags)… would you have any idea why the performance is poor? It performs no better than it would at random.

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [September 6, 2021, 7:27am UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/7 "2021-09-06T07:27:13Z")

</div>

Please create a new post. That’s a completely separate subject than this topic.

---

<div class="post-metadata">

**Author:** ![Rainiefantasy](https://avatars.discourse-cdn.com/v4/letter/r/9dc877/32.png) [@Rainiefantasy](https://discuss.huggingface.co/u/Rainiefantasy)\
**Post date:** [September 6, 2021, 2:16pm UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/8 "2021-09-06T14:16:05Z")

</div>

Ah sorry my intention of creating it here is because I wanted to ask if the contextualised word embeddings generated above are actually appropriate to use for this, but I will create a separate post,  
Thanks

---

<div class="post-metadata">

**Author:** ![Sanjeeth](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@Sanjeeth](https://discuss.huggingface.co/u/Sanjeeth)\
**Post date:** [October 4, 2021, 1:27pm UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/9 "2021-10-04T13:27:19Z")

</div>

Hi Bram Vanroy, I was trying the above mentioned code with respect to BertTokenizer instead of AutoTokenizer but i get a error as mentioned below.

**ValueError: word\_ids() is not available when using Python-based tokenizers**

Can you please let me know what changes to be made in code to get a list indicating the word corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [October 4, 2021, 1:43pm UTC](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/10 "2021-10-04T13:43:00Z")

</div>

You need to use the fast version of the tokenizer (`BertTokenizerFast`) if you want to use that functionality.