# Setting "num\_beams" and using "past\_key\_values" when calling .generate()

**URL:** <https://discuss.huggingface.co/t/setting-num-beams-and-using-past-key-values-when-calling-generate/84752>\
**Category:** 🤗Transformers\
**Created:** [May 2, 2024, 3:53am UTC](https://discuss.huggingface.co/t/setting-num-beams-and-using-past-key-values-when-calling-generate/84752 "2024-05-02T03:53:23Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![darvish](https://avatars.discourse-cdn.com/v4/letter/d/e5b9ba/32.png) [@darvish](https://discuss.huggingface.co/u/darvish)\
**Post date:** [May 2, 2024, 3:53am UTC](https://discuss.huggingface.co/t/setting-num-beams-and-using-past-key-values-when-calling-generate/84752/1 "2024-05-02T03:53:23Z")

</div>

I have a piece of code to accelerate text generation using past\_key\_values. The simplified version is as follows:

```auto
prefix_output = model(prefix_input_ids)
generation_output = model.generate(postfix_input_ids, num_beams=1, use_cache=True, past_key_values=prefix_output.past_key_values)

```

Here the variable “model” can be “GPT2LMHeadModel” that has loaded “gpt2-xl”. The code works perfectly fine. The problem is that if “num\_beams” is set to greater than 1, then I get the exception below (in the example I set “num\_beams” to 3):

> ‘Sizes of tensors must match except in dimension 2. Expected size 1 but got size 3 for tensor number 1 in the list.’

I suspect that I should somehow pre-process the values of “prefix\_output.past\_key\_values”, before passing it to model.generate(). I am not sure though. Anybody knows how to fix this? Thanks.