# Embeddings from the Decoder only model **URL:** \ **Category:** Research\ **Created:** [September 20, 2024, 1:36pm UTC](https://discuss.huggingface.co/t/embeddings-from-the-decoder-only-model/107497 "2024-09-20T13:36:59Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![manojkumar427](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/manojkumar427/32/31842_2.png) [@manojkumar427](https://discuss.huggingface.co/u/manojkumar427)\ **Post date:** [September 20, 2024, 1:36pm UTC](https://discuss.huggingface.co/t/embeddings-from-the-decoder-only-model/107497/1 "2024-09-20T13:36:59Z")
I am trying to extract the embedding from a decoder only LLM. I tried using hidden states by appending EOS token to input and passing it to model. But embeddings taken by the hidden state values of EOS in last\_hidden\_layer or concatenation of all tokens hidden state values from last\_hidden\_layer aren’t performing well using cosine similarity of different prompts. Is there any way to extract the embedding from decoder only model in order to compare the different prompts. ---
**Author:** ![mahmutc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mahmutc/32/52583_2.png) [@mahmutc](https://discuss.huggingface.co/u/mahmutc)\ **Post date:** [September 20, 2024, 2:34pm UTC](https://discuss.huggingface.co/t/embeddings-from-the-decoder-only-model/107497/2 "2024-09-20T14:34:42Z")
hi @manojkumar427 > But embeddings taken by the hidden state values of EOS in last\_hidden\_layer or concatenation of all tokens hidden state values from last\_hidden\_layer aren’t performing well using cosine similarity of different prompts. Can you please give a specific example: what is the output and what do you expect? ---
**Author:** ![MattiLinnanvuori](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MattiLinnanvuori](https://discuss.huggingface.co/u/MattiLinnanvuori)\ **Post date:** [September 22, 2024, 9:21am UTC](https://discuss.huggingface.co/t/embeddings-from-the-decoder-only-model/107497/3 "2024-09-22T09:21:48Z")
[Stack Overflow post](https://stackoverflow.com/questions/76926025/sentence-embeddings-from-llama-2-huggingface-opensource) shows several ways to do it. ---
**Author:** ![manojkumar427](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/manojkumar427/32/31842_2.png) [@manojkumar427](https://discuss.huggingface.co/u/manojkumar427)\ **Post date:** [October 8, 2024, 7:54am UTC](https://discuss.huggingface.co/t/embeddings-from-the-decoder-only-model/107497/4 "2024-10-08T07:54:27Z")
Hi @mahmutc , Thanks for your intereset, I am trying to generate vector representation of prompt using decoder only model, so input would be the prompt/sentence and output would be the vector representing prompt/sentence. We can use these vectors for comparing with other prompt/sentence. It would be a great work since decoder only models are rapidly evolving. ---
**Author:** ![manojkumar427](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/manojkumar427/32/31842_2.png) [@manojkumar427](https://discuss.huggingface.co/u/manojkumar427)\ **Post date:** [October 8, 2024, 8:09am UTC](https://discuss.huggingface.co/t/embeddings-from-the-decoder-only-model/107497/5 "2024-10-08T08:09:16Z")
Hi @MattiLinnanvuori , Thanks for the result, I tried both of them, first method(wgt avg pooling has an issue of length: resultant vector depends on length of prompt) and second method also didn’t perform well , I think paper has proper explanation. Working over it. Thanks! Great help. ---
**Author:** ![CrisTiru](https://avatars.discourse-cdn.com/v4/letter/c/a6a055/32.png) [@CrisTiru](https://discuss.huggingface.co/u/CrisTiru)\ **Post date:** [March 26, 2025, 4:36pm UTC](https://discuss.huggingface.co/t/embeddings-from-the-decoder-only-model/107497/6 "2025-03-26T16:36:43Z")
Hi, I wanna to do same job for Gemma 3 , @manojkumar427 did you find out?