# Questions about default checkpointing behavior (train v. val)

**URL:** <https://discuss.huggingface.co/t/questions-about-default-checkpointing-behavior-train-v-val/9106>\
**Category:** Beginners\
**Created:** [August 10, 2021, 12:57am UTC](https://discuss.huggingface.co/t/questions-about-default-checkpointing-behavior-train-v-val/9106 "2021-08-10T00:57:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![w-nicole](https://avatars.discourse-cdn.com/v4/letter/w/839c29/32.png) [@w-nicole](https://discuss.huggingface.co/u/w-nicole)\
**Post date:** [August 10, 2021, 12:57am UTC](https://discuss.huggingface.co/t/questions-about-default-checkpointing-behavior-train-v-val/9106/1 "2021-08-10T00:57:06Z")

</div>

Hello,

I had a few questions about how Huggingface checkpoint behavior changes depending on the arguments to the Trainer.

In the documentation, I noticed that by default:

1. because evaluation\_strategy is ‘no’, evaluation is never run during training. Reference was here: [Trainer — transformers 4.7.0 documentation](https://huggingface.co/transformers/main_classes/trainer.html#transformers.TrainingArguments)
2. because metric\_for\_best\_model is by default ‘None’, the metric defaults to “loss”, which is the same as “eval\_loss”. The same reference as above.

My questions were:

1. If I run a model with default arguments, does the model checkpointing automatically save and load the best checkpoint based on validation loss/the eval\_dataloader?

2. Do the losses displayed in trainer\_state.json correspond to val or train loss?

3. Is there an easy way to plot the train and val losses that doesn’t involve overriding the default model behavior or going through an external visualization library like comet?

Additionally, I tried manually overwriting the metric\_for\_best\_model as follows:  
training\_args.metric\_for\_best\_model = “eval\_loss”

I do this before the training arguments are passed to Trainer initialization.

If I do this, 1) is this a correct way to enforce validation-based checkpointing? and 2) in this situation, what are the losses displayed in trainer\_state.json?

Thank you for your help! I appreciate it.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 10, 2021, 8:20am UTC](https://discuss.huggingface.co/t/questions-about-default-checkpointing-behavior-train-v-val/9106/2 "2021-08-10T08:20:12Z")

</div>

Hi there, here are the answers:

1. No, by default the model checkpointing only saves model to resume training later if something goes wrong, but there is no best model loading logic unless you use `load_best_model_at_end`. You will then need to set an `eval_strategy` and a `save_strategy` that match (either epoch or steps)

2. It’s the accumulated training loss since the beginning of the training.

3. No there is not inside Trainer. We integrate with most reporting tooling, TensorBoard, Wandb, CometML etc for this reason.

For your last questions, setting `metric_for_best_model` is not enough, you need to set `load_best_model_at_end` to `True`. The losses dispalyed in the `trainer_state.json` will still be the training losses.

---

<div class="post-metadata">

**Author:** ![w-nicole](https://avatars.discourse-cdn.com/v4/letter/w/839c29/32.png) [@w-nicole](https://discuss.huggingface.co/u/w-nicole)\
**Post date:** [August 23, 2021, 3:08pm UTC](https://discuss.huggingface.co/t/questions-about-default-checkpointing-behavior-train-v-val/9106/3 "2021-08-23T15:08:47Z")

</div>

Thank you for your help!

If I have load\_best\_model\_at\_end activated when I train my model, whenever I use the

BertForMaskedLM.from\_pretrained()

am I guaranteed to automatically recover my best checkpoint?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 30, 2021, 6:21pm UTC](https://discuss.huggingface.co/t/questions-about-default-checkpointing-behavior-train-v-val/9106/4 "2021-08-30T18:21:45Z")

</div>

If you use the `from_pretrained` method you will get the model associated with the folder/model identifier you pass. This class has no knowledge of the best checkpoint.

---

<div class="post-metadata">

**Author:** ![CryyingFace](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cryyingface/32/20390_2.png) [@CryyingFace](https://discuss.huggingface.co/u/CryyingFace)\
**Post date:** [October 16, 2023, 1:54pm UTC](https://discuss.huggingface.co/t/questions-about-default-checkpointing-behavior-train-v-val/9106/5 "2023-10-16T13:54:09Z")

</div>

hi,is it possible to save model base on the best traning loss without the need to pass `eval_strategy` ?