# Metrics for Training Set in Trainer

**URL:** <https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461>\
**Category:** 🤗Transformers\
**Created:** [December 3, 2020, 10:11am UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461 "2020-12-03T10:11:12Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bumblebert](https://avatars.discourse-cdn.com/v4/letter/b/5e9695/32.png) [@Bumblebert](https://discuss.huggingface.co/u/Bumblebert)\
**Post date:** [December 3, 2020, 10:11am UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/1 "2020-12-03T10:11:12Z")

</div>

Hey guys,

I am currently using the [Trainer](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainer#trainer) in order to train my `DistilBertForSequenceClassification`.

My problem: I want to stepwise print/save the loss and accuracy of my **training set** by using the Trainer. Is there a way to do so?

What I did so far: I have adjusted `compute_metrics`. But this function is only carried out on my evaluation set. I need the same for my training set. I also tried out the [TrainerCallback](https://huggingface.co/transformers/main_classes/callback.html). But I can’t access the current predictions of the model by using the predefined callbacks.

Another idea would be to customize the Trainer using a custom train function. But firstly I want to ask you whether there is an easier way to do so.

Thank you in advance!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [December 3, 2020, 2:04pm UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/2 "2020-12-03T14:04:05Z")

</div>

There is no way to do this directly in the `Trainer`, it’s just not built that way (because evaluation is often pretty slow). You should twek the code in your own subclass of `Trainer` to add a `self.evaluate(self.train_dataset)` at the appropriate line and then handle the logging.

---

<div class="post-metadata">

**Author:** ![Bumblebert](https://avatars.discourse-cdn.com/v4/letter/b/5e9695/32.png) [@Bumblebert](https://discuss.huggingface.co/u/Bumblebert)\
**Post date:** [December 4, 2020, 12:14pm UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/3 "2020-12-04T12:14:14Z")

</div>

Thank you very much for your fast reply!  
Just to be sure that my problem was correctly described in my post above: My intention is that I can print, plot and monitor the loss and accuracy of my training set while training by using the `Trainer`. Currently only the loss of my training dataset is printed while carrying out the training with the `Trainer`. If I would use the [Fine-tuning with native PyTorch](https://huggingface.co/transformers/custom_datasets.html#ft-trainer) I can add an accuracy function in the training-loop, which also calculates the accuracy (or other metrics) on my training-set per epoch besides the loss. In the end, I want to ask whether it is easily possible to do the same with my `Trainer`.

To sum up, I dont want to newly evaluate something, because the model is predicting the labels in order to calculate the loss anyway. I just want to further use this predictions to calculate metrics on my training set.

Sorry for probably being unclear. Have a great weekend!

---

<div class="post-metadata">

**Author:** ![mbforbes](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mbforbes/32/2499_2.png) [@mbforbes](https://discuss.huggingface.co/u/mbforbes)\
**Post date:** [August 10, 2021, 5:09pm UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/4 "2021-08-10T17:09:53Z")

</div>

I know this is an old issue, but I came across this while trying to determine the best way to track metrics besides the loss during training. I thought I’d post what I came up with in case it helps someone else.

To reiterate the context, like @Bumblebert, I’m interested in running additional metrics on the outputs that the model already computes during training, rather than running an additional evaluation run over the entire training set (using, e.g., `self.evaluate(self.train_dataset)`). My use case is that I’m training a multiple choice model and I’d like to see how the accuracy changes while training.

I’ve found the suggestion in the `Trainer` class to _“Subclass and override for custom behavior.”_ to be a good idea a couple of times now 🙂 To compute custom metrics, I found where the outputs were easily accessible, in `compute_loss()`, and added some code. I’ve prefixed `MAX:` to my comments below:

```python
class CustomTrainer(Trainer):

    def compute_loss(self, model, inputs, return_outputs=False):
        """
        MAX: Subclassed to compute training accuracy.

        How the loss is computed by Trainer. By default, all models return the loss in
        the first element.

        Subclass and override for custom behavior.
        """
        if self.label_smoother is not None and "labels" in inputs:
            labels = inputs.pop("labels")
        else:
            labels = None
        outputs = model(**inputs)

        # MAX: Start of new stuff.
        if "labels" in inputs:
            preds = outputs.logits.detach()
            acc = (
                (preds.argmax(axis=1) == inputs["labels"])
                .type(torch.float)
                .mean()
                .item()
            )
            self.log({"accuracy": acc})
        # MAX: End of new stuff.

        # Save past state if it exists
        # TODO: this needs to be fixed and made cleaner later.
        if self.args.past_index >= 0:
            self._past = outputs[self.args.past_index]

        if labels is not None:
            loss = self.label_smoother(outputs, labels)
        else:
            # We don't use .loss here since the model may return tuples instead of
            # ModelOutput.
            loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]

        return (loss, outputs) if return_outputs else loss

```

Then, I instantiate a `CustomTrainer` instead of a `Trainer` and run as normal.

(Note that the above code isn’t battle-tested, and I only tried on a single GPU. So take it as a starting point and with a grain of salt.)

I started using the _wandb_ plotting integration, which is sent the results of `self.log()` that we added, and automatically makes a plot:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/c/c8e16f4537bf2dce0e48879c5d6fdc0730a97503.png)

(my runs were called `train_metrics` and `train_metrics_2`)

Cheers!

---

<div class="post-metadata">

**Author:** ![sid8491](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sid8491/32/3610_2.png) [@sid8491](https://discuss.huggingface.co/u/sid8491)\
**Post date:** [January 3, 2022, 10:05am UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/5 "2022-01-03T10:05:21Z")

</div>

I did this by adding a custom callback which calls the evaluate() method with train\_dataset at the end of every callback.

```auto
class CustomCallback(TrainerCallback):
    
    def __init__ (self, trainer) -> None:
        super(). __init__ ()
        self._trainer = trainer
    
    def on_epoch_end(self, args, state, control, **kwargs):
        if control.should_evaluate:
            control_copy = deepcopy(control)
            self._trainer.evaluate(eval_dataset=self._trainer.train_dataset, metric_key_prefix="train")
            return control_copy

trainer = Trainer(
    model=model, # the instantiated Transformers model to be trained
    args=training_args, # training arguments, defined above
    train_dataset=train_dataset, # training dataset
    eval_dataset=valid_dataset, # evaluation dataset
    compute_metrics=compute_metrics, # the callback that computes metrics of interest
    tokenizer=tokenizer
)
trainer.add_callback(CustomCallback(trainer)) 
train = trainer.train()

```

This gives the train metrics like following:

> {‘train\_loss’: 0.7159061431884766, ‘train\_accuracy’: 0.4, ‘train\_f1’: 0.5714285714285715, ‘train\_runtime’: 6.2973, ‘train\_samples\_per\_second’: 2.382, ‘train\_steps\_per\_second’: 0.159, ‘epoch’: 1.0}  
> {‘eval\_loss’: 0.8529007434844971, ‘eval\_accuracy’: 0.0, ‘eval\_f1’: 0.0, ‘eval\_runtime’: 2.0739, ‘eval\_samples\_per\_second’: 0.964, ‘eval\_steps\_per\_second’: 0.482, ‘epoch’: 1.0}

---

<div class="post-metadata">

**Author:** ![emmakelo](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@emmakelo](https://discuss.huggingface.co/u/emmakelo)\
**Post date:** [January 17, 2022, 2:37pm UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/6 "2022-01-17T14:37:42Z")

</div>

Hello,

How do you do to get the train accuracy score, I just get the train loss when using the trainer() ?.

---

<div class="post-metadata">

**Author:** ![Kaveri](https://avatars.discourse-cdn.com/v4/letter/k/82dd89/32.png) [@Kaveri](https://discuss.huggingface.co/u/Kaveri)\
**Post date:** [January 5, 2023, 2:24pm UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/7 "2023-01-05T14:24:32Z")

</div>

I will expand on @[sid8491](https://discuss.huggingface.co/u/sid8491)’s answer. In my use case, I have to keep finetuning on multiple datasets from different languages. This way, we can keep track of metrics (loss, precision, recall, …) across different language datasets.

If anyone has suggestions for cleaner code, please do suggest.

```auto
from sklearn.metrics import precision_recall_fscore_support, accuracy_score, log_loss
from torch.nn import CrossEntropyLoss
import numpy as np
from copy import deepcopy
from transformers import TrainerCallback

lang = 'en'

class CustomCallback(TrainerCallback):
    
    def __init__ (self, trainer) -> None:
        super(). __init__ ()
        self._trainer = trainer
    
    def on_epoch_end(self, args, state, control, **kwargs):
        if control.should_evaluate:
            control_copy = deepcopy(control)
            self._trainer.evaluate(eval_dataset=self._trainer.train_dataset, metric_key_prefix="train@"+lang)
            return control_copy

def compute_metrics(pred):
    global num_labels
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='macro')
    acc = accuracy_score(labels, preds)
    loss_fct = CrossEntropyLoss()
    logits = torch.tensor(pred.predictions)
    labels = torch.tensor(labels)
    loss = loss_fct(logits.view(-1, num_labels), labels.view(-1))
    return {
        'accuracy@'+lang: acc,
        'f1@'+lang: f1,
        'precision@'+lang: precision,
        'recall@'+lang: recall,
        'loss@'+lang: loss,
    }

training_args = TrainingArguments(
    evaluation_strategy="epoch",
    logging_strategy="epoch",
    learning_rate=1e-4,
    num_train_epochs=2,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    output_dir=MODEL_DIR+'_EN',
    overwrite_output_dir=True,
    remove_unused_columns=False,
    save_total_limit=1,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=en_train_dataset,
    eval_dataset=en_valid_dataset,
    compute_metrics=compute_metrics
)

trainer.add_callback(CustomCallback(trainer)) 

train_result = trainer.train()

trainer.evaluate(metric_key_prefix='test_en',
                eval_dataset=en_test_dataset)

```

---

<div class="post-metadata">

**Author:** ![qiying](https://avatars.discourse-cdn.com/v4/letter/q/8e8cbc/32.png) [@qiying](https://discuss.huggingface.co/u/qiying)\
**Post date:** [February 26, 2024, 10:31am UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/9 "2024-02-26T10:31:56Z")

</div>

Thank you. This helps me a lot

---

<div class="post-metadata">

**Author:** ![jaandoui](https://avatars.discourse-cdn.com/v4/letter/j/b4bc9f/32.png) [@jaandoui](https://discuss.huggingface.co/u/jaandoui)\
**Post date:** [April 11, 2024, 12:53pm UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/10 "2024-04-11T12:53:38Z")

</div>

That’s a good way to do it. But I guess the problem with that is that you are training, then evaluating on the training data, meaning that you are running through the data twice, instead of just collecting the results of the training.

---

<div class="post-metadata">

**Author:** ![jaandoui](https://avatars.discourse-cdn.com/v4/letter/j/b4bc9f/32.png) [@jaandoui](https://discuss.huggingface.co/u/jaandoui)\
**Post date:** [April 11, 2024, 9:11pm UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/11 "2024-04-11T21:11:41Z")

</div>

I answered this problem for getting metrics for epoch and for batch, also getting other metrics f1, …  
here [Batch and Epoch training metrics for transformers Trainer - Stack Overflow](https://stackoverflow.com/questions/78311534/batch-and-epoch-training-metrics-for-transformers-trainer/78311535#78311535)

---

<div class="post-metadata">

**Author:** ![tcm03](https://avatars.discourse-cdn.com/v4/letter/t/c0e974/32.png) [@tcm03](https://discuss.huggingface.co/u/tcm03)\
**Post date:** [February 13, 2025, 6:13am UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/12 "2025-02-13T06:13:42Z")

</div>

Hi,  
I found your solution appropriate in my case because for a large training set, it would be costly in terms of time and money to go through the entire training set the second time for evaluation.  
However, as far as I know, the `compute_loss` function is specific to a batch only (it is contained in either `training_step()` or `prediction_step()` for in-batch processing), so if we calculate the metrics in that function, the result is over that batch. For evaluating the metrics over the entire training set, how can we modify the code without going through the training set the second time as mentioned by @sgugger ?  
Thanks

---

<div class="post-metadata">

**Author:** ![pmulcaire](https://avatars.discourse-cdn.com/v4/letter/p/439d5e/32.png) [@pmulcaire](https://discuss.huggingface.co/u/pmulcaire)\
**Post date:** [March 14, 2025, 9:41pm UTC](https://discuss.huggingface.co/t/metrics-for-training-set-in-trainer/2461/13 "2025-03-14T21:41:40Z")

</div>

Very old thread now, but I found mbforbes’ solution above to work well. But compute\_loss() is called for both training and eval steps, so both will end up in your logs, which may not be what you want. To solve this my very hacky solution was to compute a hash for each eval example, save the list of such hashes in the Trainer subclass, and check against that list before computing the metric:

```auto
class CustomTrainer(Trainer):

    def __init__ (self, *args, **kwargs):
        super(). __init__ (*args, **kwargs)
        self.eval_ids = args["eval_ids"]

    def compute_loss(self, model, inputs, return_outputs=False):
        [...]
        if "labels" in inputs:
            example_id = hashlib.md5(json.dumps(inputs["input_ids"][0].tolist()).encode()).hexdigest() # get the first instance in the batch, as a representative
             if example_id not in self.eval_ids:
                 # compute and log metrics
            [...]

```

fwiw!