# Avoid recalculating hidden states between generate calls?

**URL:** <https://discuss.huggingface.co/t/avoid-recalculating-hidden-states-between-generate-calls/34209>\
**Category:** 🤗Transformers\
**Created:** [March 20, 2023, 9:01am UTC](https://discuss.huggingface.co/t/avoid-recalculating-hidden-states-between-generate-calls/34209 "2023-03-20T09:01:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shaike04](https://avatars.discourse-cdn.com/v4/letter/s/ac91a4/32.png) [@Shaike04](https://discuss.huggingface.co/u/Shaike04)\
**Post date:** [March 20, 2023, 9:01am UTC](https://discuss.huggingface.co/t/avoid-recalculating-hidden-states-between-generate-calls/34209/1 "2023-03-20T09:01:30Z")

</div>

I have a 100-token prompt which I use to generate some text. This will take X milliseconds to achieve.  
My next generate call involves the same 100 tokens as the previous call plus 3 more tokens.

To speed up the response time, I want to avoid re-calculating the hidden states for the past 100 tokens (and only process the new 3 tokens).

How can I achieve that? I can not find my way with the docs.

Thank you

---

<div class="post-metadata">

**Author:** ![kateland](https://avatars.discourse-cdn.com/v4/letter/k/77aa72/32.png) [@kateland](https://discuss.huggingface.co/u/kateland)\
**Post date:** [March 22, 2023, 1:12am UTC](https://discuss.huggingface.co/t/avoid-recalculating-hidden-states-between-generate-calls/34209/2 "2023-03-22T01:12:08Z")

</div>

You will basically need to smuggle the hidden state between generate() invocations. The hidden state is also called `past_key_values` in various parts of the code. I ended up having to take over the generation loop and slightly patch Transformers in order to do this. In very rough Python, your modification to the generation loop will look something like this:

```auto
def generate(model, old_prompt, added, past_state):
    ...
    # catch up decoding new chars since past_state
    prompt = old_prompt
    for new_token in added:
        prompt += new_token
        input = model.prepare_inputs_for_generation(prompt, **past_state)
        output = model(**input, return_dict=True)
        past_state['past_key_values'] = output['past_key_values']
    # resume generation from here after decoding is done
    while True:
        input = model.prepare_inputs_for_generation(prompt, **past_state)
        output = model(**input, return_dict=True)
        ...
    ...

```

I basically copied sample() from transformers/generation/utils.py and modified it in the way I just described. I know it’s far from complete code, but it should give you an idea as to what is required. I don’t doubt there’s an easier way to do this as well, but it didn’t seem like the Transformers generate() function allows passing in the initial hidden state (else I wouldn’t have had to write a patch like this). This should really be possible out of the box, imagine how many cycles are being wasted in people’s experiments that use Transformers for incremental text generation.

---

<div class="post-metadata">

**Author:** ![Shaike04](https://avatars.discourse-cdn.com/v4/letter/s/ac91a4/32.png) [@Shaike04](https://discuss.huggingface.co/u/Shaike04)\
**Post date:** [March 26, 2023, 8:44am UTC](https://discuss.huggingface.co/t/avoid-recalculating-hidden-states-between-generate-calls/34209/3 "2023-03-26T08:44:23Z")

</div>

Thank you. The major value of this would be the improvement in the time\token metric.  
The thing is that every change in the generate function might be not optimal and harm this metric 🙂

I will update after I manage to test it and see if it improves generation times.

---

<div class="post-metadata">

**Author:** ![Shaike04](https://avatars.discourse-cdn.com/v4/letter/s/ac91a4/32.png) [@Shaike04](https://discuss.huggingface.co/u/Shaike04)\
**Post date:** [March 30, 2023, 6:11am UTC](https://discuss.huggingface.co/t/avoid-recalculating-hidden-states-between-generate-calls/34209/4 "2023-03-30T06:11:53Z")

</div>

See [FasterTransformer/gpt\_guide.md at main · NVIDIA/FasterTransformer · GitHub](https://github.com/NVIDIA/FasterTransformer/blob/main/docs/gpt_guide.md) they have a solution for that