# The hidden\_states when i use model.generate

**URL:** <https://discuss.huggingface.co/t/the-hidden-states-when-i-use-model-generate/73169>\
**Category:** 🤗Transformers\
**Created:** [February 14, 2024, 1:50pm UTC](https://discuss.huggingface.co/t/the-hidden-states-when-i-use-model-generate/73169 "2024-02-14T13:50:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![MikeDean](https://avatars.discourse-cdn.com/v4/letter/m/51bf81/32.png) [@MikeDean](https://discuss.huggingface.co/u/MikeDean)\
**Post date:** [February 14, 2024, 1:50pm UTC](https://discuss.huggingface.co/t/the-hidden-states-when-i-use-model-generate/73169/1 "2024-02-14T13:50:22Z")

</div>

Here is my code:

```python
from transformers import MistralForCausalLM, AutoTokenizer, GenerationConfig
model = MistralForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
prompt = "[INST] What's your name? [/INST]"
generate_config = GenerationConfig(
    temperature=1,
    top_p = 0.75,
    top_k=40,
    num_beams=4,
    output_hidden_states=True
)
input_ids = tokenizer(prompt)['input_ids']
generation_output = model.generate(
    input_ids,
    generation_config=generate_config,
    return_dict_in_generate=True,
)
print(len(generation_output.hidden_states))

```

I am looking to retrieve the final layer hidden states for every token, which includes those in the input sequence, while excluding the language model head (lm\_head). How can I go about this?

---

<div class="post-metadata">

**Author:** ![Eang](https://avatars.discourse-cdn.com/v4/letter/e/977dab/32.png) [@Eang](https://discuss.huggingface.co/u/Eang)\
**Post date:** [May 31, 2024, 4:27pm UTC](https://discuss.huggingface.co/t/the-hidden-states-when-i-use-model-generate/73169/2 "2024-05-31T16:27:57Z")

</div>

Same problem here, no output

---

<div class="post-metadata">

**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\
**Post date:** [June 1, 2024, 7:16am UTC](https://discuss.huggingface.co/t/the-hidden-states-when-i-use-model-generate/73169/3 "2024-06-01T07:16:11Z")

</div>

Hi! You have to add `output_hidden_states=True` into generate to get the hidden\_states

```python
generation_output = model.generate(
    input_ids,
    generation_config=generate_config,
    return_dict_in_generate=True,
    output_hidden_states=True,
)

```

---

<div class="post-metadata">

**Author:** ![juyil](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@juyil](https://discuss.huggingface.co/u/juyil)\
**Post date:** [March 27, 2025, 11:10pm UTC](https://discuss.huggingface.co/t/the-hidden-states-when-i-use-model-generate/73169/4 "2025-03-27T23:10:55Z")

</div>

It only have output token’s hidden state, how to get hidden states for every token, which includes those in the input sequence?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 28, 2025, 3:03am UTC](https://discuss.huggingface.co/t/the-hidden-states-when-i-use-model-generate/73169/5 "2025-03-28T03:03:47Z")

</div>

Hmm… **return\_dict\_in\_generate=True**?

* * *

To retrieve the final layer hidden states for every token, including those in the input sequence, while using `model.generate()`, you can follow these steps:

1. **Set the correct configuration** : Ensure that `output_hidden_states=True` and `return_dict_in_generate=True` in your `GenerationConfig`.

2. **Use the `forward()` method**: First, pass the input tokens through the model using `model.forward()` to get the hidden states of the input sequence.

3. **Generate the output** : Then, use `model.generate()` to generate the output tokens and capture their hidden states.

Here’s the modified code:

```python
from transformers import MistralForCausalLM, AutoTokenizer, GenerationConfig
import torch

model = MistralForCausalLM.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

prompt = "[INST] What's your name? [/INST]"
input_ids = tokenizer(prompt, return_tensors="pt")["input_ids"]

# Step 1: Get hidden states for the input sequence
input_hidden_states = model(
    input_ids,
    output_hidden_states=True
).hidden_states[-1]

# Step 2: Generate with the model and get hidden states for generated tokens
generate_config = GenerationConfig(
    temperature=1,
    top_p=0.75,
    top_k=40,
    num_beams=4,
    output_hidden_states=True
)

generation_output = model.generate(
    input_ids,
    generation_config=generate_config,
    return_dict_in_generate=True,
)

with torch.no_grad():
    # Get hidden states for the generated tokens
    generated_hidden_states = generation_output.hidden_states[-1]

# Combine input and generated hidden states
hidden_states = torch.cat([input_hidden_states, generated_hidden_states], dim=1)

print(hidden_states.shape) # Output shape: (batch_size, total_tokens, hidden_size)

```

### Explanation:

1. **Get Input Hidden States** : The `model.forward()` method processes the input tokens and returns their hidden states.

2. **Generate and Capture Output Hidden States** : The `model.generate()` method generates the output tokens and returns their hidden states when `output_hidden_states=True` and `return_dict_in_generate=True`.

3. **Combine Hidden States** : Finally, we combine the hidden states of the input and generated tokens into a single tensor, giving the hidden states for every token in the entire sequence.

By following these steps, you can obtain the hidden states for every token, including those in the input sequence, while using `model.generate()`.