# \[Maybe Bug\] When using EarlyStopping Callbacks with Seq2SeqTraininer, training didn't stop

**URL:** <https://discuss.huggingface.co/t/maybe-bug-when-using-earlystopping-callbacks-with-seq2seqtraininer-training-didnt-stop/25883>\
**Category:** DeepSpeed\
**Created:** [November 10, 2022, 8:27pm UTC](https://discuss.huggingface.co/t/maybe-bug-when-using-earlystopping-callbacks-with-seq2seqtraininer-training-didnt-stop/25883 "2022-11-10T20:27:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [November 10, 2022, 8:27pm UTC](https://discuss.huggingface.co/t/maybe-bug-when-using-earlystopping-callbacks-with-seq2seqtraininer-training-didnt-stop/25883/1 "2022-11-10T20:27:56Z")

</div>

When trying to use EarlyStopping for Seq2SeqTrainer, e.g. patience was set to 1 and threshold 1.0:

```auto
training_args = Seq2SeqTrainingArguments(
    output_dir='./',
    num_train_epochs=3,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    logging_steps=1,
    save_steps=5,
    eval_steps=1,
    max_steps=10,
    evaluation_strategy="steps",
    predict_with_generate=True,
    report_to=None,
    metric_for_best_model="chr_f_score",
    load_best_model_at_end=True
)

early_stop = EarlyStoppingCallback(2, 1.0)

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=valid_data.with_format("torch"),
    eval_dataset=test_data.with_format("torch"),
    compute_metrics=compute_metrics,
    callbacks=[early_stop]
)

trainer.train()

```

The model continues training until `max_steps` instead of stopping after the stopping criteria is met.

**I’m not sure if this is a bug or maybe some argument is missing in when I use Seq2SeqTrainer** , a working code to replicate the issue can be found on [Huggingface EarlyStopping Callbacks | Kaggle](https://www.kaggle.com/code/alvations/huggingface-earlystopping-callbacks?scriptVersionId=110637297)

* * *

After the `max_steps`, if we do some probing, somehow the `early_stopping_patience_counter` has been reached but the training didn’t stop

```auto
>>> early_stop.early_stopping_patience_counter
2

```

---

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [November 10, 2022, 8:30pm UTC](https://discuss.huggingface.co/t/maybe-bug-when-using-earlystopping-callbacks-with-seq2seqtraininer-training-didnt-stop/25883/2 "2022-11-10T20:30:33Z")

</div>

Also asked on [python - Why did the Seq2SeqTrainer not stop when the EarlyStoppingCallback criteria is met? - Stack Overflow](https://stackoverflow.com/q/74394999/610569)

---

<div class="post-metadata">

**Author:** ![alvations](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/alvations/32/97_2.png) [@alvations](https://discuss.huggingface.co/u/alvations)\
**Post date:** [November 16, 2022, 11:05am UTC](https://discuss.huggingface.co/t/maybe-bug-when-using-earlystopping-callbacks-with-seq2seqtraininer-training-didnt-stop/25883/3 "2022-11-16T11:05:27Z")

</div>

Found the issue, when the `save_state` is not in pace with the `eval_steps` the patience and threshold for the early stopping works in a different way, where the patience will kick in after the first `save_state` is met.

**This might be a feature / bug.**

Setting the `eval_steps` to be the same as the `save_state` would return the early stopping behavior as we intuitively expect.

Details on [python - Why did the Seq2SeqTrainer not stop when the EarlyStoppingCallback criteria is met? - Stack Overflow](https://stackoverflow.com/a/74459350/610569)

---

<div class="post-metadata">

**Author:** ![Ikeofai](https://avatars.discourse-cdn.com/v4/letter/i/c67d28/32.png) [@Ikeofai](https://discuss.huggingface.co/u/Ikeofai)\
**Post date:** [April 4, 2024, 10:50am UTC](https://discuss.huggingface.co/t/maybe-bug-when-using-earlystopping-callbacks-with-seq2seqtraininer-training-didnt-stop/25883/4 "2024-04-04T10:50:11Z")

</div>

The problem here is that the save\_steps and eval\_steps are not equal. The model is set to evaluate at 1 step and the results printed at logging\_steps of 1, even if model is set to stop at some patience, model cannot stop until it gets to the next save\_step which you set as 5.

I faced this issue recently and I think this comment may help.

In the documentation it states that you have to set save\_steps at same value as eval\_steps in your Training arguments logic. I set my model to save at number of steps corresponding to the end of every epoch with a code save\_steps=len(dataset)//batch\_size, this gives me number of steps per epoch. This number has to be same as eval\_steps which represents the number of steps at which an evaluation on the evaluation dataset happens. That way the early stopping will kick in at right time. If not the model will keep running until it gets to the next save\_steps point specified at which it continues running if evaluation does not take place at that point. docs link here:[Callbacks](https://huggingface.co/docs/transformers/en/main_classes/callback)

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/0/5/057644512c388cf0acbccefbb5aa4e0560234a63.png)