# Trainer.evaluate() doesn't return evaluation loss

**URL:** <https://discuss.huggingface.co/t/trainer-evaluate-doesnt-return-evaluation-loss/140841>\
**Category:** 🤗Transformers\
**Created:** [February 14, 2025, 12:30pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-doesnt-return-evaluation-loss/140841 "2025-02-14T12:30:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nitishpandey04](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nitishpandey04/32/33204_2.png) [@nitishpandey04](https://discuss.huggingface.co/u/nitishpandey04)\
**Post date:** [February 14, 2025, 12:30pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-doesnt-return-evaluation-loss/140841/1 "2025-02-14T12:30:17Z")

</div>

I am using Seq2SeqTrainer to lora finetune madlad 3b model using my own dataset.

When I run `trainer.evaluate()`, the evaluation loss is not printing.

**Code:**

```auto

output_dir = "madlad_run_5"
training_args = Seq2SeqTrainingArguments(
    output_dir,
    per_device_train_batch_size=32,
    # per_device_eval_batch_size=2,
    num_train_epochs=1,
    # max_steps=1000,
    # gradient_accumulation_steps=8,
    bf16=True,
    torch_compile=True,
    # gradient_checkpointing=True,
    # torch_empty_cache_steps=10,
    learning_rate=3e-4,
    lr_scheduler_type="cosine",
    weight_decay=0.00001,
    warmup_ratio=0.1,
    report_to="tensorboard",
    logging_dir=f"{output_dir}/tensorboard_logs",
    logging_first_step=True,
    logging_steps=1,
    save_strategy="epoch",
    # save_steps=1000,
    save_total_limit=1,
    eval_strategy="steps",
    eval_steps=5,
    include_for_metrics=["loss"],
    # batch_eval_metrics=True,
    optim="paged_adamw_32bit",
    dataloader_pin_memory=True, # optimization
    dataloader_num_workers=4,
    # eval_on_start=True,
)

trainer = Seq2SeqTrainer(
    peft_model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=test_dataset, 
    data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model),
    # compute_metrics=compute_metrics,
    # callbacks=[eval_loss_callback] # Add custom callback
)

eval_results = trainer.evaluate()
print(eval_results)

# trainer.train()

# peft_model.save_pretrained("madlad_run_2_lora_ckpt")

```

**Output:**

```auto
{'eval_runtime': 10.9764, 'eval_samples_per_second': 9.11, 'eval_steps_per_second': 0.456}

```

I want to get the evaluation loss also.

**Some extra context:**  
I am using tensorboard for logging and in the initial training runs, I was not getting the evaluation loss also in the logs. So I tried some solutions like adding callbacks, `compute_metrics` function, etc but the evaluation loss was still not coming.

Then I tried to directly run `trainer.evaluate()` but here also evaluation loss was missing.

How to get the evaluation loss also when the model is evaluated during training i.e. in the evaluation logs and in `trainer.evaluate()`?

**System configuration:**

```auto
- `transformers` version: 4.48.3
- Platform: Linux-5.15.0-127-generic-x86_64-with-glibc2.35
- Python version: 3.10.16
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config: - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 3
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA A40

```

Let me know if any other information is required. Please resolve this.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [February 14, 2025, 2:02pm UTC](https://discuss.huggingface.co/t/trainer-evaluate-doesnt-return-evaluation-loss/140841/2 "2025-02-14T14:02:50Z")

</div>

It looks like it will be difficult to get a loss…

> <https://github.com/huggingface/transformers/issues/32307>
>
> \### Feature request
> 
> Hi, I am requesting the feature to make evaluation loss acc…essible inside \`compute\_metrics()\` within the Trainer class, this will enable users to log loss dependent metrics during training, in my case I want to track perplexity. By making the pre-computed validation loss available users can accurately compute these metrics without having to recompute them from scratch.
> 
> The existing solution computes perplexity score post training by running the model through a validation batch via \`trainer.evaluate()\`, this only provides a single score on a single validation batch after all training is done.
> 
> 
> \### Motivation
> 
> Why other solutions to track perplexity during training via \`compute\_metrics\` do not work effectively :
> 
> \- Using \`evaluate\` library to load perplexity : when using Trainer, it \[reloads\](https://github.com/huggingface/evaluate/blob/main/metrics/perplexity/perplexity.py#L103) the model from the hub at each evaluation which is inefficient. Furthermore, the implementation is not that flexible, for instance it assumes that \`labels\` are exactly equal to \`input\_ids\` which is not always the case, for instance I’m running instruction finetuning where the labels are the response part of the \`input\_ids\`.
> 
> \- Computing perplexity direclty using \`EvalPrediction\`'s logits and labels to compute the loss again inside of \`compute\_metrics\` function would be inaccurate since we do not have access to \`attention\_mask\` of the validation batches, so we can’t compute proper loss by masking padding tokens.
> 
> \- Why make loss accessible in \`compute\_metrics\` along with logits and labels :
> - The evaluation loss is computed by default earlier in the evaluation loop, and \`compute\_metrics\` is executed right after.
> - I’m assured that the loss used is accurate and consistent and thus the reported perplexity is reliable. Furthermore, the loss can be used to compute other metrics other than perplexity.
> - I believe this feature would be useful since there are similar request In the community like \[this\](https://discuss.huggingface.co/t/compute-perplexity-using-compute-metrics-in-sfttrainer/74974) and \[this\](https://discuss.huggingface.co/t/using-perplexity-as-metric-during-training/42354).
> 
> \### Your contribution
> 
> I would be happy to implement this and get your feedback on whether it is the right approach before opening PR 😊.

---

<div class="post-metadata">

**Author:** ![nitishpandey04](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nitishpandey04/32/33204_2.png) [@nitishpandey04](https://discuss.huggingface.co/u/nitishpandey04)\
**Post date:** [February 17, 2025, 7:49am UTC](https://discuss.huggingface.co/t/trainer-evaluate-doesnt-return-evaluation-loss/140841/3 "2025-02-17T07:49:22Z")

</div>

I see. But I wonder why is such a basic thing as “evaluation loss” so difficult to get with Trainer? Why the design is like that? Is it not important?

I think eval loss is extremely important to track to understand model overfitting right? We can use that to perform early stopping too, but none of this is possible with Trainer.

I’m curious why the design is like that. Any thought?