# Ask for help: Output inconsistency when using LLM batch inference compared to single input

**URL:** <https://discuss.huggingface.co/t/ask-for-help-output-inconsistency-when-using-llm-batch-inference-compared-to-single-input/146303>\
**Category:** Beginners\
**Created:** [March 18, 2025, 3:09pm UTC](https://discuss.huggingface.co/t/ask-for-help-output-inconsistency-when-using-llm-batch-inference-compared-to-single-input/146303 "2025-03-18T15:09:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Skyuan](https://avatars.discourse-cdn.com/v4/letter/s/67e7ee/32.png) [@Skyuan](https://discuss.huggingface.co/u/Skyuan)\
**Post date:** [March 18, 2025, 3:09pm UTC](https://discuss.huggingface.co/t/ask-for-help-output-inconsistency-when-using-llm-batch-inference-compared-to-single-input/146303/1 "2025-03-18T15:09:27Z")

</div>

I found single LLM input get different output logits when merging into a batch for inference.

Besides, I need to use inputs\_embeds as model input.

My test LLM is “Qwen/Qwen2.5-1.5B-Instruct” and the test code is below.

```auto
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# load model and tokenizezr
model_name = "Qwen/Qwen2.5-1.5B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# set model eval
model.eval()

# input texts
texts = ['a', 'b', 'c']

# tokenize
inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True).to(model.device)

# get inputs_embeds
with torch.no_grad():
    inputs_embeds = model.get_input_embeddings()(inputs.input_ids)

# get attention_mask and position_ids
attention_mask = inputs.attention_mask
position_ids = torch.arange(inputs.input_ids.shape[1], device=model.device).unsqueeze(0).expand(inputs.input_ids.shape[0], -1)

# batch
with torch.no_grad():
    output_batch = model(
        inputs_embeds=inputs_embeds,
        attention_mask=attention_mask,
        position_ids=position_ids
    ).logits[0]

# single
with torch.no_grad():
    output_single = model(
        inputs_embeds=inputs_embeds[0].unsqueeze(0), 
        attention_mask=attention_mask[0].unsqueeze(0),
        position_ids=position_ids[0].unsqueeze(0)
    ).logits[0] 

# check consistency
is_close = torch.allclose(output_batch, output_single, atol=1e-5, rtol=1e-3)
print(is_close)

```

I tried all the methods Deepseek suggested and then all failed, like setting attention masks, positions and so on.

I want the same output logits of a single input text as the ones extracted from batch output results.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 18, 2025, 3:58pm UTC](https://discuss.huggingface.co/t/ask-for-help-output-inconsistency-when-using-llm-batch-inference-compared-to-single-input/146303/2 "2025-03-18T15:58:28Z")

</div>

Perhaps KV-cache issue?

> [@Transformer KV-Cache Produces Worse Output Than Normal Generation – Why?](https://discuss.huggingface.co/t/transformer-kv-cache-produces-worse-output-than-normal-generation-why/143839):
>
> Hi everyone, I implemented a transformer model from scratch based on Andrej Karpathy’s video. After that, I added some improvements, such as parallelized multi-head attention and KV-Cache for efficient generation. However, I am facing a problem: when I enable KV-Cache, the generated text is significantly worse than when KV-Cache is disabled. I expected KV-Cache to improve speed while keeping output quality the same, but my implementation seems to degrade output quality. I suspect there might…

> <https://github.com/huggingface/transformers/issues/30670>
>
> \### System Info
> 
> transformers.\_\_version\_\_ = 4.41.0.dev0
> Python 3.10.12
> unix
> …
> 
> \### Who can help?
> 
> @gante
> 
> \### Information
> 
> \- \[X\] The official example scripts
> \- \[\] My own modified scripts
> 
> \### Tasks
> 
> \- \[x\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> When using the code documented \[here\](https://huggingface.co/docs/transformers/main/en/llm\_optims?static-kv=Static+Cache#static-kv-cache-and-torchcompile), we get different generations for the same prompt when generating it in a different batch.
> 
> For example if you run the code bellow 1st generation for the prompt hey is different than the 2nd generation.
> 
> \`\`\`python
> import os
> from transformers import LlamaTokenizer, LlamaForCausalLM
> from transformers import StaticCache
> import torch
> 
> 
> def decode\_one\_tokens(model, cur\_token, input\_pos, cache\_position, past\_key\_values):
> logits = model(
> cur\_token,
> position\_ids=input\_pos,
> cache\_position=cache\_position,
> past\_key\_values=past\_key\_values,
> return\_dict=False,
> use\_cache=True
> )\[0\]
> new\_token = torch.argmax(logits\[:, -1\], dim=-1)\[:, None\]
> return new\_token
> 
> 
> def generate(prompts: list\[str\], model: LlamaForCausalLM, tokenizer: LlamaTokenizer, num\_tokens\_to\_generate: int = 40) -\> list\[str\]:
> global decode\_one\_tokens
> inputs = tokenizer(prompts, return\_tensors="pt", padding=True).to(model.device)
> batch\_size, seq\_length = inputs\["input\_ids"\].shape
> with torch.no\_grad():
> past\_key\_values = StaticCache(
> config=model.config, max\_batch\_size=batch\_size, max\_cache\_len=4096, device=torch\_device, dtype=model.dtype
> )
> cache\_position = torch.arange(seq\_length, device=torch\_device)
> generated\_ids = torch.zeros(
> batch\_size, seq\_length + num\_tokens\_to\_generate + 1, dtype=torch.int, device=torch\_device
> )
> generated\_ids\[:, cache\_position\] = inputs\["input\_ids"\].to(torch\_device).to(torch.int)
> with torch.backends.cuda.sdp\_kernel(enable\_flash=False, enable\_mem\_efficient=False, enable\_math=True):
> logits = model(
> \*\*inputs, cache\_position=cache\_position, past\_key\_values=past\_key\_values,return\_dict=False, use\_cache=True
> )\[0\]
> next\_token = torch.argmax(logits\[:, -1\], dim=-1)\[:, None\]
> generated\_ids\[:, seq\_length\] = next\_token\[:, 0\]
>         
> # Not using torch.compile to simplify debugging
> # decode\_one\_tokens = torch.compile(decode\_one\_tokens, mode="reduce-overhead", fullgraph=True)
> cache\_position = torch.tensor(\[seq\_length + 1\], device=torch\_device)
> for \_ in range(1, num\_tokens\_to\_generate):
> with torch.backends.cuda.sdp\_kernel(enable\_flash=False, enable\_mem\_efficient=False, enable\_math=True):
> next\_token = decode\_one\_tokens(model, next\_token.clone(), None, cache\_position, past\_key\_values)
> generated\_ids\[:, cache\_position\] = next\_token.int()
> cache\_position += 1
> 
> return tokenizer.batch\_decode(generated\_ids, skip\_special\_tokens=True)
> 
> 
> if \_\_name\_\_ == "\_\_main\_\_":
> torch\_device = "cuda"
> tokenizer = LlamaTokenizer.from\_pretrained("meta-llama/Llama-2-7b-hf", pad\_token="\</s\>", padding\_side="right")
> model = LlamaForCausalLM.from\_pretrained("meta-llama/Llama-2-7b-hf", torch\_dtype=torch.float16).to(torch\_device)
> model.eval()
> print(generate(\["hey", "yo"\], model, tokenizer)\[0\])
> # prints: hey, i'm alex.\\ni'm a 20-something year old photographer and filmmaker based in los angeles. i'm a lover of all things cre
> print('-'\*50 + '\\n'\*2)
> print(generate(\["hey", "yo code math"\], model, tokenizer)\[0\])
> #prints: hey The 2018-19 season is the 10th season of the National Women's Soccer League (NWSL), the top division of women's soccer
> print('-'\*50 + '\\n'\*2)
> \`\`\`
> 
> \### Expected behavior
> 
> The generation for each prompt should not depend on the other examples in the batch.

And Tips by Hugging Chat

* * *

To address the inconsistency in logits between single and batch inputs when using `inputs_embeds`, ensure that the `inputs_embeds` match the model’s data type. Convert `inputs_embeds` to the model’s `torch_dtype` before inference. Modify the code as follows:

```python
# get inputs_embeds
with torch.no_grad():
    inputs_embeds = model.get_input_embeddings()(inputs.input_ids)
    # Ensure inputs_embeds are in the correct dtype
    if model.config.torch_dtype is not None:
        inputs_embeds = inputs_embeds.to(model.config.torch_dtype)

```

This converts the embeddings to the model’s expected dtype, ensuring consistency between single and batch inference.

**Answer:**  
ensure that `inputs_embeds` are converted to the model’s `torch_dtype` before inference. Modify the code by adding the dtype conversion step:

```python
# Add this line after getting inputs_embeds
inputs_embeds = inputs_embeds.to(model.config.torch_dtype)

```

This adjustment ensures that the data types are consistent between batch and single inputs, resolving the inconsistency issue [2].

---

<div class="post-metadata">

**Author:** ![Skyuan](https://avatars.discourse-cdn.com/v4/letter/s/67e7ee/32.png) [@Skyuan](https://discuss.huggingface.co/u/Skyuan)\
**Post date:** [March 19, 2025, 3:18am UTC](https://discuss.huggingface.co/t/ask-for-help-output-inconsistency-when-using-llm-batch-inference-compared-to-single-input/146303/3 "2025-03-19T03:18:16Z")

</div>

Thanks for your help!  
I tried the methods mentioned the above posts, including setting “use\_cache=False”, manually setting attention mask and making sure of the same dype, but all failed.  
I further found that only “cuda” causes the inconsistency problem and “cpu” works fine, but still struggle to make “cuda” batch inference procuce consistent results.  
The numerical gap is kind of big:  
tensor([[7.5312, 9.3750, 6.0625, …, -3.6250, -3.6250, -3.6250]],  
device=‘cuda:0’) # batch  
tensor([[7.2812, 9.2500, 6.2188, …, -3.7969, -3.7969, -3.7969]],  
device=‘cuda:0’) # single

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 19, 2025, 4:56am UTC](https://discuss.huggingface.co/t/ask-for-help-output-inconsistency-when-using-llm-batch-inference-compared-to-single-input/146303/4 "2025-03-19T04:56:37Z")

</div>

It seems to happen with anything other than **torch.float32** , and it seems to be particularly noticeable with **torch.bfloat16**. There are also some who point out that it is a unique problem with Qwen 2.5.  
With bfloat16, Attention may also be suspicious.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# load model and tokenizezr
model_name = "Qwen/Qwen2.5-1.5B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
).eval().to(torch.float32) # if bfloat16, it causes inconsistency
tokenizer = AutoTokenizer.from_pretrained(model_name)

print("model.dtype: ", model.dtype)
print("model.device: ", model.device)

# input texts
texts = ['a', 'b', 'c']

# tokenize
inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True).to(model.device)

# get inputs_embeds
with torch.no_grad():
    inputs_embeds = model.get_input_embeddings()(inputs.input_ids)

# get attention_mask and position_ids
attention_mask = inputs.attention_mask
position_ids = torch.arange(inputs.input_ids.shape[1], device=model.device).unsqueeze(0).expand(inputs.input_ids.shape[0], -1)

# batch
with torch.no_grad():
    output_batch = model(
        inputs_embeds=inputs_embeds,
        attention_mask=attention_mask,
        position_ids=position_ids
    ).logits[0]

# single
with torch.no_grad():
    output_single = model(
        inputs_embeds=inputs_embeds[0].unsqueeze(0), 
        attention_mask=attention_mask[0].unsqueeze(0),
        position_ids=position_ids[0].unsqueeze(0)
    ).logits[0] 

# check consistency
is_close = torch.allclose(output_batch, output_single, atol=1e-5, rtol=1e-3)
print("consistent?: ", is_close)
print("batch: ", output_batch)
print("single: ", output_single)

```

### torch.bfloat16

```auto
model.dtype: torch.bfloat16
model.device: cuda:0
  attn_output = torch.nn.functional.scaled_dot_product_attention(
consistent?: False
batch: tensor([[7.4688, 9.3125, 6.0625, ..., -3.5469, -3.5469, -3.5469]],
       device='cuda:0', dtype=torch.bfloat16)
single: tensor([[6.9375, 8.9375, 5.9375, ..., -3.7188, -3.7188, -3.7188]],
       device='cuda:0', dtype=torch.bfloat16)

```

### torch.float32

```auto
model.dtype: torch.float32
model.device: cuda:0
consistent?: False
batch: tensor([[7.6105, 9.9338, 6.7679, ..., -3.6860, -3.6860, -3.6861]],
       device='cuda:0')
single: tensor([[7.6105, 9.9338, 6.7679, ..., -3.6860, -3.6860, -3.6861]],
       device='cuda:0')

```

### Possible causes

[https://github.com/QwenLM/Qwen2.5-VL/issues/706](https://github.com/QwenLM/Qwen2.5-VL/issues/706)

> <https://github.com/unslothai/unsloth/issues/1456>
>
> I am experiencing an issue with batch inference using my self-trained model. Whe…n I perform inference on single samples, the results are consistent and correct. However, when I perform inference on batches of multiple samples, the results differ unexpectedly.
> 
> I also find it strange that the outputs of batch inference change when I alter the batch size. I’ve tested batch sizes ranging from 8 to 64, and the inconsistencies increase with larger batch sizes.
> 
> I've updated the unsloth version to 2024.12.4 and also set padding\_side to 'left' and set tokenizer.pad\_token = tokenizer.unk\_token, it still not work.
> 
> Here is my code
> \`\`\`
> max\_seq\_length = 1024 # in case of truncate 
> dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
> load\_in\_4bit = True # Use 4bit quantization to reduce memory usage. Can be False.
> 
> 
> model, tokenizer = FastLanguageModel.from\_pretrained(
> model\_name = f"{weights\_path}", # YOUR MODEL YOU USED FOR TRAINING
> max\_seq\_length = max\_seq\_length,
> dtype = dtype,
> load\_in\_4bit = load\_in\_4bit,
> )
> FastLanguageModel.for\_inference(model) 
> 
> tokenizer.padding\_side='left'
> tokenizer.pad\_token = tokenizer.unk\_token
> 
> batch\_size = 4 #32 #64   
> 
> \# Prepare the batch
> batch\_input\_strs = \[\]
> batch\_data\_js\_items = \[\]
> 
> \# Iterate through data and prepare inputs in batches
> for idx, data\_js\_item in enumerate(data\_js\[:eval\_item\_num\]):
> input\_str = f"game\_record:{data\_js\_item\['game\_record'\]}, 'target\_player':{data\_js\_item\['target\_player'\]}"
> batch\_input\_strs.append(input\_str)
> batch\_data\_js\_items.append(data\_js\_item)
> 
> # Once the batch size is reached, or we've processed the last item
> if len(batch\_input\_strs) == batch\_size or idx == len(data\_js\[:eval\_item\_num\]) - 1:
> # Prepare batch inputs for tokenizer
> inputs = tokenizer(
> \[alpaca\_prompt.format(
> INSTRCTION, input\_str, "",) for input\_str in batch\_input\_strs
> \],
> return\_tensors="pt",padding=True, truncation=True).to("cuda")
> # Perform batch inference
> outputs = model.generate(\*\*inputs, max\_new\_tokens=1024, use\_cache=True, do\_sample = False)
> 
> # Decode the batch outputs
> output\_lst = tokenizer.batch\_decode(outputs)
> 
> for ouput\_token in output\_lst:
> ouput\_token = ouput\_token.replace(tokenizer.pad\_token, "")
>         
> # Process results for each item in the batch
> for i, output\_text in enumerate(output\_lst):
> # Extract the response text from the model output
> s\_idx = output\_text.find("### Response:\\n") + len("### Response:\\n")
> e\_idx = output\_text.find(EOS\_TOKEN)
> predict\_str = output\_text\[s\_idx:e\_idx\]
>         
> batch\_input\_strs = \[\]
> batch\_data\_js\_items = \[\]
> 
> \`\`\`

[https://huggingface.co/docs/transformers/model\_doc/auto#transformers.FlaxAutoModelForMaskedLM.from\_config.attn\_implementation](https://huggingface.co/docs/transformers/model_doc/auto#transformers.FlaxAutoModelForMaskedLM.from_config.attn_implementation)

> **[GPU](https://huggingface.co/docs/transformers/perf_infer_gpu_one)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![Skyuan](https://avatars.discourse-cdn.com/v4/letter/s/67e7ee/32.png) [@Skyuan](https://discuss.huggingface.co/u/Skyuan)\
**Post date:** [March 20, 2025, 3:47am UTC](https://discuss.huggingface.co/t/ask-for-help-output-inconsistency-when-using-llm-batch-inference-compared-to-single-input/146303/5 "2025-03-20T03:47:08Z")

</div>

Thanks for your suggestions, and I finally decide to use float32.

In addition, in this case of quantized LLMs, turning model to float32 still gives inconsistent outputs, perhaps because quantized LLM has its special mechanism for some mathematical operations which can’t be transfered to float32 model by simply changing its parametres’ dtype. (model.dequantize() raises some NotImplementedError when using Qwen2.5-1.5B-Instruct) I compromise by decomposing the batch to single inputs and bearing the low efficiency.