# Model.generate use\_cache=True generates different results than use\_cache=False

**URL:** <https://discuss.huggingface.co/t/model-generate-use-cache-true-generates-different-results-than-use-cache-false/144001>\
**Category:** Intermediate\
**Created:** [March 4, 2025, 11:34am UTC](https://discuss.huggingface.co/t/model-generate-use-cache-true-generates-different-results-than-use-cache-false/144001 "2025-03-04T11:34:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![edenlum](https://avatars.discourse-cdn.com/v4/letter/e/e8c25b/32.png) [@edenlum](https://discuss.huggingface.co/u/edenlum)\
**Post date:** [March 4, 2025, 11:34am UTC](https://discuss.huggingface.co/t/model-generate-use-cache-true-generates-different-results-than-use-cache-false/144001/1 "2025-03-04T11:34:04Z")

</div>

Using huggingface transformers library, I see different outputs when generating text with model.generate with and without use\_cache argument. Is this intended and how can I combat this?  
The scores when I use cache (from the second token generated and onwards) are different. AFAIK use\_cache is an optimization that shouldn’t effect the outputs. I also got this error on gpu (here in the code I use 'cpu). Code to reproduce:

MODEL = “meta-llama/Meta-Llama-3.1-8B-Instruct”

from transformers import AutoTokenizer, AutoModelForCausalLM, DataCollatorWithPadding  
import torch

# MODEL = “Qwen/Qwen2.5-0.5B-Instruct”

MODEL = “meta-llama/Meta-Llama-3.1-8B-Instruct”

tokenizer = AutoTokenizer.from\_pretrained(MODEL, padding\_side=‘left’)  
model = AutoModelForCausalLM.from\_pretrained(MODEL, torch\_dtype=torch.bfloat16) # , cache\_dir=“/workspace/fmrai/”)

tokenizer.pad\_token\_id = tokenizer.eos\_token\_id  
tokenizer.pad\_token = tokenizer.eos\_token

model.eval()

device = ‘cpu’ # ‘cuda’ if torch.cuda.is\_available() else ‘cpu’  
model.to(device)

# Example text to generate from

prompt = “Tell me something that is very exciting”

# Format the prompt using the chat template

formatted\_prompt = tokenizer.apply\_chat\_template([{“role”: “user”, “content”: prompt}], tokenize=False)

# Tokenize the input

inputs = tokenizer(formatted\_prompt, return\_tensors=“pt”, padding=True)

def generate(model, tokenizer, inputs, use\_cache, output\_attentions=False, device=‘cpu’):  
with torch.no\_grad():  
outputs = model.generate(  
inputs[“input\_ids”].to(device),  
attention\_mask=inputs[“attention\_mask”].to(device),  
max\_new\_tokens=2,  
pad\_token\_id=tokenizer.eos\_token\_id,  
do\_sample=False,  
return\_dict\_in\_generate=True,  
use\_cache=use\_cache,  
output\_scores=True,  
# output\_hidden\_states=True,  
output\_attentions=output\_attentions,  
)  
return outputs

outputs\_cache = generate(model, tokenizer, inputs, use\_cache=True, device=device)  
outputs\_no\_cache = generate(model, tokenizer, inputs, use\_cache=False, device=device)  
outputs\_cache\_attentions = generate(model, tokenizer, inputs, use\_cache=True, output\_attentions=True, device=device)  
outputs\_no\_cache\_attentions = generate(model, tokenizer, inputs, use\_cache=False, output\_attentions=True, device=device)

for i in range(2):  
print(f"Cache {i}: {outputs\_cache.scores[i]}“)  
print(f"No Cache {i}: {outputs\_no\_cache.scores[i]}”)  
print(f"Cache Att {i}: {outputs\_cache\_attentions.scores[i]}“)  
print(f"No Cache Att {i}: {outputs\_no\_cache\_attentions.scores[i]}”)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 4, 2025, 1:24pm UTC](https://discuss.huggingface.co/t/model-generate-use-cache-true-generates-different-results-than-use-cache-false/144001/2 "2025-03-04T13:24:50Z")

</div>

Similar case?

> [@Transformer KV-Cache Produces Worse Output Than Normal Generation – Why?](https://discuss.huggingface.co/t/transformer-kv-cache-produces-worse-output-than-normal-generation-why/143839):
>
> Hi everyone, I implemented a transformer model from scratch based on Andrej Karpathy’s video. After that, I added some improvements, such as parallelized multi-head attention and KV-Cache for efficient generation. However, I am facing a problem: when I enable KV-Cache, the generated text is significantly worse than when KV-Cache is disabled. I expected KV-Cache to improve speed while keeping output quality the same, but my implementation seems to degrade output quality. I suspect there might…

---

<div class="post-metadata">

**Author:** ![edenlum](https://avatars.discourse-cdn.com/v4/letter/e/e8c25b/32.png) [@edenlum](https://discuss.huggingface.co/u/edenlum)\
**Post date:** [March 4, 2025, 1:51pm UTC](https://discuss.huggingface.co/t/model-generate-use-cache-true-generates-different-results-than-use-cache-false/144001/3 "2025-03-04T13:51:31Z")

</div>

This case talks about a problem with his custom implementation. I am facing a problem with the official model.generate function in huggingface transformers.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 4, 2025, 2:55pm UTC](https://discuss.huggingface.co/t/model-generate-use-cache-true-generates-different-results-than-use-cache-false/144001/4 "2025-03-04T14:55:59Z")

</div>

I found a cause.

> <https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535>
>
> \### System Info
> 
> transformers==4.31.0
> 
> \- huggingface\_hub version: 0.15.1
> \- …Platform: Linux-5.15.0-78-generic-x86\_64-with-glibc2.35
> \- Python version: 3.10.12
> \- Running in iPython ?: No
> \- Running in notebook ?: No
> \- Running in Google Colab ?: No
> \- Token path ?: /u/k/h/khanov/.cache/huggingface/token
> \- Has saved token ?: False
> \- Configured git credential helpers:
> \- FastAI: N/A
> \- Tensorflow: N/A
> \- Torch: 2.0.0
> \- Jinja2: 3.0.3
> \- Graphviz: N/A
> \- Pydot: N/A
> \- Pillow: 9.0.1
> \- hf\_transfer: N/A
> \- gradio: N/A
> \- numpy: 1.24.2
> \- ENDPOINT: https://huggingface.co
> \- HUGGINGFACE\_HUB\_CACHE: /u/k/h/khanov/.cache/huggingface/hub
> \- HUGGINGFACE\_ASSETS\_CACHE: /u/k/h/khanov/.cache/huggingface/assets
> \- HF\_TOKEN\_PATH: /u/k/h/khanov/.cache/huggingface/token
> \- HF\_HUB\_OFFLINE: False
> \- HF\_HUB\_DISABLE\_TELEMETRY: False
> \- HF\_HUB\_DISABLE\_PROGRESS\_BARS: None
> \- HF\_HUB\_DISABLE\_SYMLINKS\_WARNING: False
> \- HF\_HUB\_DISABLE\_EXPERIMENTAL\_WARNING: False
> \- HF\_HUB\_DISABLE\_IMPLICIT\_TOKEN: False
> \- HF\_HUB\_ENABLE\_HF\_TRANSFER: False
> 
> \### Who can help?
> 
> @ArthurZucker, @younesbelkada
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[X\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[X\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> I was working on a custom decoding method, however, I found a deviation from greedy search when using KV caching.
> 
> \`\`\`python
> import torch
> import transformers
> from transformers import AutoTokenizer, AutoModelForCausalLM
> from tqdm import tqdm
> 
> MODEL\_PATH = "/nobackup-fast/khanov/llama-7b" # "huggyllama/llama-7b"
> GEN\_DEV = "cuda:0"
> 
> tokenizer = AutoTokenizer.from\_pretrained(MODEL\_PATH)
> model = AutoModelForCausalLM.from\_pretrained(MODEL\_PATH, torch\_dtype=torch.bfloat16).to(GEN\_DEV)
> 
> def get\_input\_ids(prompt: str) -\> torch.Tensor:
> global model, tokenizer
> tokens = tokenizer(prompt, return\_tensors="pt").input\_ids.to(GEN\_DEV)
> return tokens
> def tokens\_to\_text(tokens: torch.Tensor):
> return tokenizer.batch\_decode(tokens, skip\_special\_tokens=True)
> 
> 
> PROMPT = "This is a " # this is just a test prompt
> 
> \# greedy decoding without caching 
> tokens = get\_input\_ids(PROMPT)
> for \_ in tqdm(range(40)):
> with torch.no\_grad():
> mout = model(tokens)
> tokens = torch.hstack((tokens, torch.argmax(mout.logits\[0, -1\]).unsqueeze(0).unsqueeze(0)))
>     
> without\_cache = tokens\_to\_text(tokens)\[0\]
> print(f"{without\_cache=}")
> 
> \# greedy decoding WITH caching 
> tokens = get\_input\_ids(PROMPT)
> cached = None
> for \_ in tqdm(range(40)):
> with torch.no\_grad():
> if cached is None:
> mout = model(tokens, output\_hidden\_states=True, use\_cache=True)
> cached = mout.past\_key\_values
> else:
> mout = model(tokens, past\_key\_values=cached, use\_cache=True, output\_hidden\_states=True)
> cached = mout.past\_key\_values
> tokens = torch.hstack((tokens, torch.argmax(mout.logits\[0, -1\]).unsqueeze(0).unsqueeze(0)))
>     
> with\_cache = tokens\_to\_text(tokens)\[0\]
> print(f"{with\_cache=}")
> 
> \# normal greedy search with HF Generate implementation
> tokens = get\_input\_ids(PROMPT)
> tokens = model.generate(tokens, num\_return\_sequences=1, max\_new\_tokens=40)
> generate\_output = tokens\_to\_text(tokens)\[0\]
> print(f"{generate\_output=}")
> 
> \# this matches exactly
> assert without\_cache == generate\_output
> 
> \# this does not!
> assert without\_cache == with\_cache
> \`\`\`
> 
> \### Expected behavior
> 
> I was expecting the results to not change when using the past\_key\_values kwarg, however, when passing past\_key\_values, the model assigned different logits to the tokens. This deviates from the model.generate behavior too. This is possibly related to #18809, and #21080.

```py
#model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16) # , cache_dir="/workspace/fmrai/")
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.float32) # , cache_dir="/workspace/fmrai/")

```

```auto
Cache 0: tensor([[3.6974, 22.4915, 14.4676, ..., -5.7953, -3.8699, -7.3202]],
       device='cuda:0')
No Cache 0: tensor([[3.6974, 22.4915, 14.4676, ..., -5.7953, -3.8699, -7.3202]],
       device='cuda:0')
Cache Att 0: tensor([[3.6974, 22.4915, 14.4676, ..., -5.7953, -3.8699, -7.3202]],
       device='cuda:0')
No Cache Att 0: tensor([[3.6974, 22.4915, 14.4676, ..., -5.7953, -3.8699, -7.3202]],
       device='cuda:0')
Cache 1: tensor([[7.8056, 6.0996, -1.6019, ..., -3.3531, 4.3285, 1.0036]],
       device='cuda:0')
No Cache 1: tensor([[7.8056, 6.0996, -1.6019, ..., -3.3530, 4.3285, 1.0037]],
       device='cuda:0')
Cache Att 1: tensor([[7.8056, 6.0996, -1.6019, ..., -3.3530, 4.3285, 1.0037]],
       device='cuda:0')
No Cache Att 1: tensor([[7.8056, 6.0996, -1.6019, ..., -3.3530, 4.3285, 1.0037]],
       device='cuda:0')

```