# Get each generated token last layer hidden state

**URL:** <https://discuss.huggingface.co/t/get-each-generated-token-last-layer-hidden-state/145921>\
**Category:** 🤗Transformers\
**Created:** [March 15, 2025, 9:19pm UTC](https://discuss.huggingface.co/t/get-each-generated-token-last-layer-hidden-state/145921 "2025-03-15T21:19:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![juyil](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@juyil](https://discuss.huggingface.co/u/juyil)\
**Post date:** [March 15, 2025, 9:19pm UTC](https://discuss.huggingface.co/t/get-each-generated-token-last-layer-hidden-state/145921/1 "2025-03-15T21:19:30Z")

</div>

I am using llama2 casual model

```auto
        multimodal_embeddings, multimodal_attention_mask = self._build_multimodal_attention(
            input_embeddings, projected_patch_embeddings, attention_mask
self.language_model.generate(inputs_embeds=multimodal_embeddings,max_new_tokens=8,output_hidden_states=True,return_dict_in_generate=True)

```

I want to get each generated token last layer hidden state.  
But I don’t know whether language\_model\_output.hidden\_states[0][-1]. is the first generated token hidden state because it is different:

```auto
language_model_output.hidden_states[0][-1].shape
torch.Size([1, 535, 4096]) # why it is same with multimodal_embeddings.shape, not 1 
(Pdb) language_model_output.hidden_states[1][-1].shape
torch.Size([1, 1, 4096])
(Pdb) multimodal_embeddings.shape
torch.Size([1, 535, 4096]) 

```

---

<div class="post-metadata">

**Author:** ![juyil](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@juyil](https://discuss.huggingface.co/u/juyil)\
**Post date:** [March 15, 2025, 9:56pm UTC](https://discuss.huggingface.co/t/get-each-generated-token-last-layer-hidden-state/145921/2 "2025-03-15T21:56:18Z")

</div>

Same problem as [Wrong shape of hidden\_states and attentions when generating · Issue #26174 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/26174)  
From this, we may need generate one more token? [Wrong shape of last layer hidden states when generating · Issue #30036 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/30036#issuecomment-2056766856)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 16, 2025, 10:34am UTC](https://discuss.huggingface.co/t/get-each-generated-token-last-layer-hidden-state/145921/3 "2025-03-16T10:34:30Z")

</div>

Hmm, that’s difficult…

> **[How to retrieve hidden states for all time steps in LSTM or BiLSTM?](https://discuss.pytorch.org/t/how-to-retrieve-hidden-states-for-all-time-steps-in-lstm-or-bilstm/1087/16)**
>
> for a fixed length of sequence, say X=\[x1,x2,…,x1000\] if you use lstm\_output, lstm\_states = nn.LSTM(X) it may cost 1 seconds while, if you use LSTMCell and for loop，it will cost you about 10 seconds, amost 10x slower than nn.LSTM. Here comes...

> **[Model outputs](https://huggingface.co/docs/transformers/main/main_classes/output#transformers.modeling_outputs.BaseModelOutput.last_hidden_state)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![juyil](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@juyil](https://discuss.huggingface.co/u/juyil)\
**Post date:** [March 16, 2025, 3:35pm UTC](https://discuss.huggingface.co/t/get-each-generated-token-last-layer-hidden-state/145921/4 "2025-03-16T15:35:16Z")

</div>

Why it is difficult?  
` language_model_output.hidden_states[1][-1].shape torch.Size([1, 1, 4096])` I think it is hidden state, right?  
Can we grep the first token hidden state by ` language_model_output.hidden_states[0][-1][:,-1,:]` ?