# \[Trainer\] Evaluation loss changes with batch size

**URL:** <https://discuss.huggingface.co/t/trainer-evaluation-loss-changes-with-batch-size/161980>\
**Category:** 🤗Transformers\
**Created:** [July 7, 2025, 12:14am UTC](https://discuss.huggingface.co/t/trainer-evaluation-loss-changes-with-batch-size/161980 "2025-07-07T00:14:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ba144220](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@ba144220](https://discuss.huggingface.co/u/ba144220)\
**Post date:** [July 7, 2025, 12:14am UTC](https://discuss.huggingface.co/t/trainer-evaluation-loss-changes-with-batch-size/161980/1 "2025-07-07T00:14:13Z")

</div>

Hi all,

I’ve been evaluating a few causal LMs (e.g. `Qwen/Qwen2.5-3B`) on 512 samples from the `togethercomputer/RedPajama-Data-1T-Sample` pertrain dataset, and I noticed that **eval loss consistently decreases as I increase the batch size** :

| Batch size | Eval loss |
| --- | --- |
| 1 | 2.414 |
| 2 | 2.340 |
| 4 | 2.299 |
| 8 | 2.298 |
| 16 | 2.296 |

I saw the same trend across other models as well. This is the code I’m using:

```python
    sft_config = SFTConfig(
        output_dir="./results",
        per_device_train_batch_size=args.batch_size,
        per_device_eval_batch_size=args.batch_size,
        dataset_text_field="text",
        max_seq_length=args.max_seq_length,
    )
    trainer = SFTTrainer(
        model=model,
        args=sft_config,
        train_dataset=dataset,
        eval_dataset=dataset,
        processing_class=tokenizer,
    )
    eval_result = trainer.evaluate()
    print(eval_result)

```

Digging in, I found that `fixed_cross_entropy` (in `loss_utils.py`) does a **token-level sum** then divides by the total non-padding token count (micro-averaging). By contrast, I implemented a **sample-wise average** (macro-averaging):

```python
# Hugging Face: token-sum / total_tokens
loss = F.cross_entropy(..., reduction="sum") / num_items_in_batch

# My version: per-sequence average then mean across sequences
loss = F.cross_entropy(..., reduction="none")
loss = loss.view(B, -1).sum(dim=1) / token_counts_per_seq
loss = loss.mean()

```

With macro-averaging, **eval loss is identical across batch sizes and input orderings** , enabling a few nice benefits:

1. We can choose optimal batch size to speed up evaluation, especially when comparing models of different sizes.
2. Sorting samples by length before batching reduces padding, halving evaluation time.

So I’m wondering:

1. Is the Trainer’s default (micro-averaging) behavior on purpose—to tie loss scale strictly to total token count?
2. Does this have any documented effect on training stability or convergence when you vary batch size?
3. Are there recommended best practices for loss normalization in large-batch LLM training (e.g. should I always override this to macro-average)?

I’d love to hear from anyone who’s dug into this or has empirical experience with different loss-averaging schemes in the 🤗Trainer. Thanks in advance!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [July 7, 2025, 4:18am UTC](https://discuss.huggingface.co/t/trainer-evaluation-loss-changes-with-batch-size/161980/2 "2025-07-07T04:18:30Z")

</div>

It seems easier to use micro-averaging in relation to gradient accumulation and multi-GPU. Other than that, there should be no problems.

> **[Fixing Gradient Accumulation](https://huggingface.co/blog/gradient_accumulation)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![ba144220](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@ba144220](https://discuss.huggingface.co/u/ba144220)\
**Post date:** [July 7, 2025, 4:43am UTC](https://discuss.huggingface.co/t/trainer-evaluation-loss-changes-with-batch-size/161980/3 "2025-07-07T04:43:11Z")

</div>

Agree. It seems like there should be two different loss implementations for evaluation and training.