# GPT2Model model output inconsistency between different transformers versions

**URL:** <https://discuss.huggingface.co/t/gpt2model-model-output-inconsistency-between-different-transformers-versions/146833>\
**Category:** Intermediate\
**Created:** [March 21, 2025, 5:36pm UTC](https://discuss.huggingface.co/t/gpt2model-model-output-inconsistency-between-different-transformers-versions/146833 "2025-03-21T17:36:35Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wenzhong2005](https://avatars.discourse-cdn.com/v4/letter/w/b3f665/32.png) [@Wenzhong2005](https://discuss.huggingface.co/u/Wenzhong2005)\
**Post date:** [March 21, 2025, 5:36pm UTC](https://discuss.huggingface.co/t/gpt2model-model-output-inconsistency-between-different-transformers-versions/146833/1 "2025-03-21T17:36:35Z")

</div>

We fine-tuned the GPT2Model (distilgpt2) some time ago. Due to tool vulnerability issues, we have to upgrade transformers 4.48.0 or above. However, the exact same GPT2 model produces different outputs for the exact same input after the upgrading. It seems to me that the masked portion of the model output changed, while the unmasked portion stays the same. Therefore, after applying a classification head (linear layer) on top of GPT-2 output, we got different scores for the same input. Can anyone help to point to what’s changed?

The code to reproduce the results:  
import torch  
import tokenizers  
import transformers  
from transformers import GPT2Model, GPT2Tokenizer

# Sample input

tokenizer = GPT2Tokenizer.from\_pretrained(“distilgpt2”)  
tokenizer.pad\_token = tokenizer.eos\_token  
tokenizer.padding\_side = “left”

text = ‘Model output changed’  
model\_inputs = tokenizer(text, padding=‘max\_length’, max\_length=12,  
truncation=True, return\_tensors=“pt”)  
input\_ids, attention\_mask = model\_inputs[“input\_ids”], model\_inputs[“attention\_mask”]  
print(‘input\_ids:’, input\_ids)  
print(‘mask:’, attention\_mask)

# Load GPT-2 Model

model = GPT2Model.from\_pretrained(“distilgpt2”)  
model.eval()

# Run model

with torch.no\_grad():  
outputs = model(input\_ids=input\_ids, attention\_mask=attention\_mask)

last\_hidden\_state = outputs.last\_hidden\_state  
print(last\_hidden\_state)

Here are the 2 requirements.txt files and model outputs:  
Before:  
torch==2.4.0  
transformers==4.41.0  
huggingface\_hub==0.27.1

input\_ids: tensor([[50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 17633, 5072, 3421]])  
mask: tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]])  
Model output:  
tensor([[[-0.1352, 0.0991, -0.2160, …, -0.1755, -0.0512, -0.0338],  
[-0.5171, -0.0978, -0.3561, …, -0.3091, 0.1552, -0.1503],  
[-0.4233, -0.1778, -0.1415, …, -0.0925, 0.1203, -0.1014],  
…,  
[-0.3410, 0.2196, -0.1369, …, -0.4246, 0.3772, -0.4357],  
[-0.6979, 0.1779, -1.0862, …, -0.5422, 0.1065, -0.2090],  
[-0.5766, 0.1015, -0.2526, …, -1.4290, -0.1708, 0.1124]]])

After:  
torch==2.4.0  
transformers==4.42.0  
huggingface\_hub==0.27.1

input\_ids: tensor([[50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 50256, 17633, 5072, 3421]])  
mask: tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1]])  
Model output:  
tensor([[[-5.1260e-02, 1.1421e-01, -6.7051e-02, …, -8.8936e-02,  
-7.6510e-02, 8.6264e-03],  
[-1.5280e-01, -5.6395e-02, 2.1665e-01, …, 1.1190e-01,  
2.2004e-02, -9.5938e-02],  
[-1.1987e-01, -5.4886e-02, 2.0053e-01, …, 1.3524e-01,  
-4.1297e-04, -8.2952e-02],  
…,  
[-3.4099e-01, 2.1960e-01, -1.3687e-01, …, -4.2462e-01,  
3.7722e-01, -4.3574e-01],  
[-6.9789e-01, 1.7786e-01, -1.0862e+00, …, -5.4218e-01,  
1.0647e-01, -2.0897e-01],  
[-5.7657e-01, 1.0148e-01, -2.5263e-01, …, -1.4290e+00,  
-1.7080e-01, 1.1240e-01]]])

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 21, 2025, 6:31pm UTC](https://discuss.huggingface.co/t/gpt2model-model-output-inconsistency-between-different-transformers-versions/146833/2 "2025-03-21T18:31:21Z")

</div>

Possibly related this phenomenon.

> [@Ask for help: Output inconsistency when using LLM batch inference compared to single input](https://discuss.huggingface.co/t/ask-for-help-output-inconsistency-when-using-llm-batch-inference-compared-to-single-input/146303):
>
> I found single LLM input get different output logits when merging into a batch for inference. Besides, I need to use inputs\_embeds as model input. My test LLM is “Qwen/Qwen2.5-1.5B-Instruct” and the test code is below. from transformers import AutoModelForCausalLM, AutoTokenizer import torch # load model and tokenizezr model\_name = "Qwen/Qwen2.5-1.5B-Instruct" model = AutoModelForCausalLM.from\_pretrained( model\_name, torch\_dtype="auto", device\_map="auto", trust\_remote\_code=Tr…

Also, the part that has changed a lot recently is the KV cache-related area, which seems to have changed quite a bit.

---

<div class="post-metadata">

**Author:** ![Wenzhong2005](https://avatars.discourse-cdn.com/v4/letter/w/b3f665/32.png) [@Wenzhong2005](https://discuss.huggingface.co/u/Wenzhong2005)\
**Post date:** [March 21, 2025, 8:36pm UTC](https://discuss.huggingface.co/t/gpt2model-model-output-inconsistency-between-different-transformers-versions/146833/3 "2025-03-21T20:36:15Z")

</div>

Thanks @John6666 for your input. I tried and it did not work. They were trying to resolve the model output inconsistency between batch run and single run, but my issue is the model output inconsistency between different transformers versions (4.39.2 vs 4.48.0). Also, the inconsistency lies in the masked portion only, but not in the unmasked portion.

---

<div class="post-metadata">

**Author:** ![Wenzhong2005](https://avatars.discourse-cdn.com/v4/letter/w/b3f665/32.png) [@Wenzhong2005](https://discuss.huggingface.co/u/Wenzhong2005)\
**Post date:** [March 21, 2025, 10:23pm UTC](https://discuss.huggingface.co/t/gpt2model-model-output-inconsistency-between-different-transformers-versions/146833/4 "2025-03-21T22:23:17Z")

</div>

After digging into it a little deeper, I found that the model output inconsistency was introduced between transformers v4.41.0 and v4.42.0.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 22, 2025, 4:55am UTC](https://discuss.huggingface.co/t/gpt2model-model-output-inconsistency-between-different-transformers-versions/146833/5 "2025-03-22T04:55:00Z")

</div>

Perhaps this? SDPA is now default attention.

> <https://github.com/huggingface/transformers/commit/b275a410057b282495422a4dcf5782418aa484e6>
>
> \* \`gpt2\` sdpa support
> 
> \* fix (at least) one test, style, repo consistency
> 
> \*… fix sdpa mask in forward --\> fixes generation
> 
> \* test
> 
> \* test2
> 
> \* test3
> 
> \* test4
> 
> \* simplify shapes for attn mask creation and small comments
> 
> \* hub fail test
> 
> \* benchmarks
> 
> \* flash attn 2 mask should not be inverted on enc-dec setup
> 
> \* fix comment
> 
> \* apply some suggestion from code review
> 
> \- only save \_attn\_implentation once
> \- remove unnecessary comment
> 
> \* change elif logic
> 
> \* \[run-slow\] gpt2
> 
> \* modify \`test\_gpt2\_sample\_max\_time\` to follow previous assertion patterns

> **[History for src/transformers/models - huggingface/transformers](https://github.com/huggingface/transformers/commits/v4.42.0/src/transformers/models)**
>
> 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX. - History for src/transformers/models - huggingface/transformers

---

<div class="post-metadata">

**Author:** ![Wenzhong2005](https://avatars.discourse-cdn.com/v4/letter/w/b3f665/32.png) [@Wenzhong2005](https://discuss.huggingface.co/u/Wenzhong2005)\
**Post date:** [March 22, 2025, 6:25pm UTC](https://discuss.huggingface.co/t/gpt2model-model-output-inconsistency-between-different-transformers-versions/146833/6 "2025-03-22T18:25:57Z")

</div>

Really appreciate your help @John6666. It worked after I switched back to the “eager” attention with attn\_implementation=“eager”.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [March 23, 2025, 6:26am UTC](https://discuss.huggingface.co/t/gpt2model-model-output-inconsistency-between-different-transformers-versions/146833/7 "2025-03-23T06:26:30Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.