# Early stopping callback problem

**URL:** <https://discuss.huggingface.co/t/early-stopping-callback-problem/5649>\
**Category:** Beginners\
**Created:** [April 21, 2021, 7:54am UTC](https://discuss.huggingface.co/t/early-stopping-callback-problem/5649 "2021-04-21T07:54:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dbejarano31](https://avatars.discourse-cdn.com/v4/letter/d/919ad9/32.png) [@dbejarano31](https://discuss.huggingface.co/u/dbejarano31)\
**Post date:** [April 21, 2021, 7:54am UTC](https://discuss.huggingface.co/t/early-stopping-callback-problem/5649/1 "2021-04-21T07:54:04Z")

</div>

Hello,

I am having problems with the EarlyStoppingCallback I set up in my trainer class as below:

```
training_args = TrainingArguments(
    output_dir = 'BERT',
    num_train_epochs = epochs,
    do_train = True,
    do_eval = True,
    evaluation_strategy = 'epoch',
    logging_strategy = 'epoch',
    per_device_train_batch_size = batch_size,
    per_device_eval_batch_size = batch_size,
    warmup_steps = 250,
    weight_decay = 0.01,
    fp16 = True,
    metric_for_best_model = 'eval_loss',
    load_best_model_at_end = True
)

trainer = MyTrainer(
    model = bert,
    args = training_args,
    train_dataset = train_dataset,
    eval_dataset = val_dataset,
    compute_metrics = compute_metrics,
    callbacks = [EarlyStoppingCallback(early_stopping_patience = 3)]
)

trainer.train()

```

I keep getting the following error:

 ![Screen Shot 2021-04-21 at 09.45.06](https://us1.discourse-cdn.com/hellohellohello/original/2X/6/6775de589846064b9d34863b378b8d226c6c2a07.png)

I already tried running the code without the `metric_for_best_model` arg, but it still gives me the same error.

I tweaked the Trainer class a bit to report metrics during training, and also created custom\_metrics to report during validation. I suspect that maybe I made a mistake there and that’s why I can’t retrieve the validation loss now. [See here](https://discuss.huggingface.co/t/logging-training-accuracy-using-trainer-class/5524) for the tweaked code.

Thanks in advance!!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [April 21, 2021, 12:09pm UTC](https://discuss.huggingface.co/t/early-stopping-callback-problem/5649/2 "2021-04-21T12:09:30Z")

</div>

You won’t be able to use the `EarlyStoppingCallback` with a nested dictionary of metrics as you did, no. And is will need the metric you are looking for to be prefixed by `eval_` (otherwise it will add it unless you change the code too). You probably will need to write your own version of the callback for this use case.

At some point, instead of rewriting the whole `Trainer`, you might be interested in writing your own training loop with [Accelerate](https://github.com/huggingface/accelerate). You can still have mixed precision training and distributed training but will have full control over your training loop. There is one example for each task using accelerate (the `run_xxx_no_trainer`) in the [examples](https://github.com/huggingface/transformers) of Transformers

---

<div class="post-metadata">

**Author:** ![dbejarano31](https://avatars.discourse-cdn.com/v4/letter/d/919ad9/32.png) [@dbejarano31](https://discuss.huggingface.co/u/dbejarano31)\
**Post date:** [April 22, 2021, 1:43pm UTC](https://discuss.huggingface.co/t/early-stopping-callback-problem/5649/3 "2021-04-22T13:43:37Z")

</div>

Thanks so much @sgugger! Will try it out!