# Getting the same embedding from llama 2 class token for any input

**URL:** <https://discuss.huggingface.co/t/getting-the-same-embedding-from-llama-2-class-token-for-any-input/60965>\
**Category:** 🤗Transformers\
**Created:** [November 3, 2023, 3:23pm UTC](https://discuss.huggingface.co/t/getting-the-same-embedding-from-llama-2-class-token-for-any-input/60965 "2023-11-03T15:23:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![oran-sh](https://avatars.discourse-cdn.com/v4/letter/o/a4c791/32.png) [@oran-sh](https://discuss.huggingface.co/u/oran-sh)\
**Post date:** [November 3, 2023, 3:23pm UTC](https://discuss.huggingface.co/t/getting-the-same-embedding-from-llama-2-class-token-for-any-input/60965/1 "2023-11-03T15:23:22Z")

</div>

Hello,  
I’m trying to extract an embedding for several sentences from llama2-7b, and I see that I am getting the same embedding vector for the class token from the last hidden layer, no matter what the input is.

```auto
from transformers import LlamaTokenizer, LlamaModel
llama2_tokenizer = LlamaTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
llama2_model = LlamaModel.from_pretrained("meta-llama/Llama-2-7b-hf")

inputs = llama2_tokenizer("put any sentence here", return_tensors="pt")
outputs = llama2_model(**inputs, return_dict=True)
print(outputs.last_hidden_state[0,0])

```

The class token is the first token, hence I index from [0,0] for a batch of size 1.  
Would appreciate your thoughts on this. Thanks!

---

<div class="post-metadata">

**Author:** ![tianyudu](https://avatars.discourse-cdn.com/v4/letter/t/5e9695/32.png) [@tianyudu](https://discuss.huggingface.co/u/tianyudu)\
**Post date:** [December 4, 2023, 8:32pm UTC](https://discuss.huggingface.co/t/getting-the-same-embedding-from-llama-2-class-token-for-any-input/60965/2 "2023-12-04T20:32:33Z")

</div>

What happened to me was that the LLAMA tokenizer added a `<SOS>` token to the front of each sentence, the `outputs.last_hidden_state[0,0]` you are querying is indeed the hidden state for the `<SOS>` token and it’s the same no matter what sentence you are using. Probably you should try `outputs.last_hidden_state[:,1]` to get the hidden state of the first token?