# Logging training accuracy using Trainer class

**URL:** <https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524>\
**Category:** 🤗Transformers\
**Created:** [April 15, 2021, 12:06pm UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524 "2021-04-15T12:06:36Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![dbejarano31](https://avatars.discourse-cdn.com/v4/letter/d/919ad9/32.png) [@dbejarano31](https://discuss.huggingface.co/u/dbejarano31)\
**Post date:** [April 15, 2021, 12:06pm UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524/1 "2021-04-15T12:06:36Z")

</div>

Hello,

I am running BertForSequenceClassification and I would like to log the accuracy as well as other metrics that I have already defined for my **training set**. I saw in another issue that I have to add a `self.evaluate(self.train_dataset)` somewhere in the code, but I am a beginner when it comes to Python and deep learning in general so I am not sure where exactly I have to include it.

I was trying to replicate the evaluate() method of the Trainer class, taking the train\_dataset as argument, but it did not work. It would really mean a lot if you could guide me as for where I should tweak the code!

Thank you for your help!

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [April 15, 2021, 4:59pm UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524/2 "2021-04-15T16:59:02Z")

</div>

hey @dbejarano31, assuming that you want to log the training metrics _during_ training, i think there are (at least) two options:

1. subclass `TrainerCallback` ([docs](https://huggingface.co/transformers/main_classes/callback.html?highlight=callbacks#trainercallback)) to create a custom callback that logs the training metrics by triggering an event with `on_evaluate`
2. subclass `Trainer` and override the `evaluate` function ([docs](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Trainer.evaluate)) to inject the additional evaluation code

option 2 might be easier to implement since you can use the existing logic as a template 🙂

---

<div class="post-metadata">

**Author:** ![dbejarano31](https://avatars.discourse-cdn.com/v4/letter/d/919ad9/32.png) [@dbejarano31](https://discuss.huggingface.co/u/dbejarano31)\
**Post date:** [April 16, 2021, 2:31pm UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524/3 "2021-04-16T14:31:40Z")

</div>

Thanks so much @lewtun!

I believe I managed to tweak the `evaluate()` method, but now I am struggling to log the metrics inside `on_evaluate()`.

I keep getting the following error:

 ![Screen Shot 2021-04-16 at 16.26.04](https://us1.discourse-cdn.com/hellohellohello/original/2X/b/b6671ed0c7fef8ca8d29977e0857808ddf671096.png)

But when I inspect the log\_history, I have both the training metrics and the eval\_loss for the first epoch. I have been trying to find `metrics` but I haven’t had any success. I suspect it must be because of how I customized `evaluate()` to output a dictionary with the validation and training metrics, so below you can find my code.

`class MyTrainer(Trainer):  
def **init** (self, model,  
args = None,  
data\_collator = None,  
train\_dataset = None,  
eval\_dataset = None,  
tokenizer = None,  
model\_init = None,  
compute\_metrics = None,  
callbacks = None,  
optimizers = (None,None)  
):

```
super(). __init__ (model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init,
              compute_metrics, callbacks, optimizers) 

```

def evaluate(  
self,  
train\_dataset = None,  
eval\_dataset: Optional[Dataset] = None,  
ignore\_keys: Optional[List[str]] = None,  
metric\_key\_prefix: str = “eval”,  
) → Dict[str, float]:

```
    # memory metrics - must set up as early as possible
    self._memory_tracker.start()

    if eval_dataset is not None and not isinstance(eval_dataset, collections.abc.Sized):
        raise ValueError("eval_dataset must implement __len__")

    train_dataloader = self.get_train_dataloader()
    eval_dataloader = self.get_eval_dataloader(eval_dataset)
    start_time = time.time()

    train_output = self.prediction_loop(
        train_dataloader,
        description = 'Training',
        prediction_loss_only = True if self.compute_metrics is None else None,
        ignore_keys = ignore_keys,
        metric_key_prefix = 'train',
        )

    eval_output = self.prediction_loop(
        eval_dataloader,
        description="Evaluation",
        # No point gathering the predictions if there are no metrics, otherwise we defer to
        # self.args.prediction_loss_only
        prediction_loss_only=True if self.compute_metrics is None else None,
        ignore_keys=ignore_keys,
        metric_key_prefix=metric_key_prefix,
    )
    train_n_samples = len(self.train_dataset)
    train_output.metrics.update(speed_metrics('train', start_time, train_n_samples))
    self.log(train_output.metrics)

    eval_n_samples = len(eval_dataset if eval_dataset is not None else self.eval_dataset)
    eval_output.metrics.update(speed_metrics(metric_key_prefix, start_time, eval_n_samples))
    self.log(eval_output.metrics)

    if self.args.tpu_metrics_debug or self.args.debug:
        # tpu-comment: Logging debug metrics for PyTorch/XLA (compile, execute times, ops, etc.)
        xm.master_print(met.metrics_report())

    self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, eval_output.metrics)
    self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, train_output.metrics)

    self._memory_tracker.stop_and_update_metrics(train_output.metrics)
    self._memory_tracker.stop_and_update_metrics(eval_output.metrics)

    dic = {
    'Training metrics': train_output.metrics,
    'Validation metrics': eval_output.metrics
    }

    return dic`

```

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [April 16, 2021, 4:13pm UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524/4 "2021-04-16T16:13:02Z")

</div>

hmm that is odd indeed. the error seems to be coming from `NotebookProgressCallback` which expects `metrics` to have an `eval_loss` field: [transformers/notebook.py at 02f7c2fe66cf3ef11402adc3d9d8a3ddd189c717 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/02f7c2fe66cf3ef11402adc3d9d8a3ddd189c717/src/transformers/utils/notebook.py#L318)

as a dirty hack, what happens if you do the following before the `on_evaluate` step is called:

```auto
eval_output.metrics["eval_loss"] = "No log"
self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, eval_output.metrics)
self.control = self.callback_handler.on_evaluate(self.args, self.state, self.control, train_output.metrics)

```

this ensures the eval metrics have an entry for `eval_loss` before the callback is called. if this hack works, it might be a bug in the callback that we should fix 🙂

---

<div class="post-metadata">

**Author:** ![dbejarano31](https://avatars.discourse-cdn.com/v4/letter/d/919ad9/32.png) [@dbejarano31](https://discuss.huggingface.co/u/dbejarano31)\
**Post date:** [April 16, 2021, 7:53pm UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524/6 "2021-04-16T19:53:46Z")

</div>

@lewtun Tried the hack and it worked!! Thanks so much for your help!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [April 17, 2021, 1:23pm UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524/7 "2021-04-17T13:23:08Z")

</div>

> [@lewtun](#):
>
> it might be a bug in the callback that we should fix

More like an oversight since this callback predates the “metric\_key\_prefix” so at that time it was impossible to have anything else than `eval_loss` 😉

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [April 17, 2021, 2:34pm UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524/8 "2021-04-17T14:34:01Z")

</div>

thanks for the context! would you agree that the callback could do with an upgrade to cover this case? (i’m happy to do it)

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [April 19, 2021, 12:52pm UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524/9 "2021-04-19T12:52:38Z")

</div>

If you want to have a go, by all means!

---

<div class="post-metadata">

**Author:** ![lixsh6](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lixsh6/32/3346_2.png) [@lixsh6](https://discuss.huggingface.co/u/lixsh6)\
**Post date:** [December 2, 2021, 8:32am UTC](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524/10 "2021-12-02T08:32:20Z")

</div>

I do agree [lewtun](https://discuss.huggingface.co/u/lewtun)’s comment. The outputs of training\_step function can only be accessed in compute\_loss function. Inside compute\_loss, loss = outputs[0] but other indices in outputs are not used. Sometimes we wish to have other metrics (e.g., training acc) including loss as the outputs and print those metrics on tensorboard. But this version can not access the outputs during training.