# Obtaining word-embeddings from Roberta **URL:** \ **Category:** Beginners\ **Created:** [June 30, 2021, 11:41am UTC](https://discuss.huggingface.co/t/obtaining-word-embeddings-from-roberta/7735 "2021-06-30T11:41:42Z")\ **Posts on this page:** 1\ **Showing post:** 9
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [July 1, 2021, 7:38am UTC](https://discuss.huggingface.co/t/obtaining-word-embeddings-from-roberta/7735/9 "2021-07-01T07:38:28Z")
First two rules of research: if something is common does not mean that it is SOTA, and what is SOTA in one task is not SOTA in another. We are still seeing feature-based systems outperform LMs in some cases (esp. low resource). It _might_ work on your task. It might even work well. If your model is specifically finetuned on single words, then it should be fine. But using a pretrained model as-is (without finetuning) to input a single word and get its embedding… I am hesitant to recommend that for the reason discussed above (most LMs are context sensitive so it is senseless to get context free representations out of it). Instead I would recommend word2vec/GloVe. In a previous post I wrote how you can extract the embeddings from a given word in an input sentence by averaging the subword logits. [Generate raw word embeddings using transformer models like BERT for downstream process - Beginners - Hugging Face Forums](https://discuss.huggingface.co/t/generate-raw-word-embeddings-using-transformer-models-like-bert-for-downstream-process/2958/2) HTIH --- _[View the full topic](https://discuss.huggingface.co/t/obtaining-word-embeddings-from-roberta/7735)._