# Why past\_key\_values is not in GreedySearchDecoderOnlyOutput?

**URL:** <https://discuss.huggingface.co/t/why-past-key-values-is-not-in-greedysearchdecoderonlyoutput/23843>\
**Category:** 🤗Transformers\
**Created:** [September 30, 2022, 6:55pm UTC](https://discuss.huggingface.co/t/why-past-key-values-is-not-in-greedysearchdecoderonlyoutput/23843 "2022-09-30T18:55:13Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![shunzh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shunzh/32/4246_2.png) [@shunzh](https://discuss.huggingface.co/u/shunzh)\
**Post date:** [September 30, 2022, 6:55pm UTC](https://discuss.huggingface.co/t/why-past-key-values-is-not-in-greedysearchdecoderonlyoutput/23843/1 "2022-09-30T18:55:13Z")

</div>

I am using a `model = GPT2LMHeadModel()` for generation. In my use case, I’ll need to call `model.generate()` for multiple times, and the `input_ids` have a shared prefix.

In my understanding, I could pass `past_key_values` as an argument in `model.generate()` so that it wouldn’t repeatedly compute the key, values of the shared prefix. However, how do I get this `past_key_values`? The `generate()` function returns a `GreedySearchDecoderOnlyOutput` object (I set beam size = 1, no sampling), which does not contain `past_key_values`. It’s only in the return of `model.forward()`?

So I’m wondering what would be a typical example of using `past_key_values` in multiple calls of `model.generate()`. Thanks!

---

<div class="post-metadata">

**Author:** ![shunzh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shunzh/32/4246_2.png) [@shunzh](https://discuss.huggingface.co/u/shunzh)\
**Post date:** [October 4, 2022, 10:55pm UTC](https://discuss.huggingface.co/t/why-past-key-values-is-not-in-greedysearchdecoderonlyoutput/23843/2 "2022-10-04T22:55:03Z")

</div>

Never mind. There’s a PR for it.

> <https://github.com/huggingface/transformers/pull/17574>
>
> \# What does this PR do?
> 
> Allows returning \`past\_key\_values\` from \`generate\` when… \`use\_cache=True\`.
> 
> Like other returned values, \`past\_key\_values\` are also returned as \`Tuple\`, one element per generated token.
> 
> Fixes #17016