# Embeddings from llama2

**URL:** <https://discuss.huggingface.co/t/embeddings-from-llama2/57306>\
**Category:** 🤗Transformers\
**Created:** [October 3, 2023, 11:46pm UTC](https://discuss.huggingface.co/t/embeddings-from-llama2/57306 "2023-10-03T23:46:41Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saaira](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@Saaira](https://discuss.huggingface.co/u/Saaira)\
**Post date:** [October 3, 2023, 11:46pm UTC](https://discuss.huggingface.co/t/embeddings-from-llama2/57306/1 "2023-10-03T23:46:41Z")

</div>

Hello,

I am trying to get sentence embeddings from a llama2 model. I tried using the feature extraction pipeline and expect the output to be a tensor of size (seq\_len, embedding\_dim). but it is a list(list(list)))

Seems like it is of size (seq\_len, vocab\_size)? Could you please help me understand why?

Or what is the right way to get a sentence embedding for a Llama model. Thanks!

```auto
from transformers import LlamaTokenizer, LlamaForCausalLM, pipeline
sentences = ["This is me", "A 2nd sentence"]
model_base_name = "meta-llama/Llama-2-7b-hf"
model = LlamaForCausalLM.from_pretrained(model_base_name)
tokenizer = LlamaTokenizer.from_pretrained(model_base_name)
feature_extraction = pipeline('feature-extraction', model=model, tokenizer=tokenizer)
embeddings = feature_extraction(sentences) # output should be of size (seq_len, embedding_dim) but is of size (seq_len, vocab_size)

(Pdb) len(embeddings[0][0][0])
32000

(Pdb) len(embeddings[0][0])
4

(Pdb) len(embeddings[0])
1

len(tokenizer)
32000

```

---

<div class="post-metadata">

**Author:** ![jasperlp](https://avatars.discourse-cdn.com/v4/letter/j/a4c791/32.png) [@jasperlp](https://discuss.huggingface.co/u/jasperlp)\
**Post date:** [October 11, 2023, 1:54am UTC](https://discuss.huggingface.co/t/embeddings-from-llama2/57306/2 "2023-10-11T01:54:44Z")

</div>

I have the same situation as mentioned by Saaira.  
Does anyone have any solutions or explanation for this?  
Thanks!

---

<div class="post-metadata">

**Author:** ![jasperlp](https://avatars.discourse-cdn.com/v4/letter/j/a4c791/32.png) [@jasperlp](https://discuss.huggingface.co/u/jasperlp)\
**Post date:** [October 11, 2023, 2:29am UTC](https://discuss.huggingface.co/t/embeddings-from-llama2/57306/3 "2023-10-11T02:29:54Z")

</div>

Hi @Saaira,

Just found the why and how for this question.

Why:  
The pipeline generally returns the first available tensor, which refers to the logits in the Llama model  
Ref:

1. [pipeline source code](https://github.com/huggingface/transformers/blob/v4.34.0/src/transformers/pipelines/feature_extraction.py#L7)
2. [Llama doc](https://huggingface.co/docs/transformers/model_doc/llama2#transformers.LlamaModel)

How:  
Instead of using the pipeline for efficiency and neat codes,  
use

```auto
model(torch.IntTensor([tokenizer(sentences)['input_ids'][0]]),return_dict=True, output_hidden_states=True)['hidden_states']

```

you can get the hidden states from all the layers (including the embedding layer) for each token,  
you will get for the first sentence

```auto
len(embeddings['hidden_states']), embeddings['hidden_states'][0].shape
(33, torch.Size([1, 4, 4096]))

```

---

<div class="post-metadata">

**Author:** ![Saaira](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@Saaira](https://discuss.huggingface.co/u/Saaira)\
**Post date:** [October 12, 2023, 12:12am UTC](https://discuss.huggingface.co/t/embeddings-from-llama2/57306/4 "2023-10-12T00:12:19Z")

</div>

Thanks @jasperlp Very helpful!

What did you find to be the best pooling strategy with llama embeddings?

---

<div class="post-metadata">

**Author:** ![jasperlp](https://avatars.discourse-cdn.com/v4/letter/j/a4c791/32.png) [@jasperlp](https://discuss.huggingface.co/u/jasperlp)\
**Post date:** [October 12, 2023, 10:16pm UTC](https://discuss.huggingface.co/t/embeddings-from-llama2/57306/5 "2023-10-12T22:16:50Z")

</div>

Haven’t tried too much on that. I suppose this is tasks by task thing

---

<div class="post-metadata">

**Author:** ![SeanLee97](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/seanlee97/32/20779_2.png) [@SeanLee97](https://discuss.huggingface.co/u/SeanLee97)\
**Post date:** [November 1, 2023, 2:05am UTC](https://discuss.huggingface.co/t/embeddings-from-llama2/57306/6 "2023-11-01T02:05:30Z")

</div>

You can use AnglE-LLaMA to extract sentence embedding from LLaMA/LLaMA2: [GitHub - SeanLee97/AnglE: Angle-optimized Text Embeddings | 🔥 New SOTA](https://github.com/SeanLee97/AnglE)

---

<div class="post-metadata">

**Author:** ![wilfoderek](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wilfoderek/32/20375_2.png) [@wilfoderek](https://discuss.huggingface.co/u/wilfoderek)\
**Post date:** [December 13, 2023, 6:43pm UTC](https://discuss.huggingface.co/t/embeddings-from-llama2/57306/7 "2023-12-13T18:43:25Z")

</div>

What is the max input length?