# Hyperparameter optimization and load\_best\_model\_at\_end

**URL:** <https://discuss.huggingface.co/t/hyperparameter-optimization-and-load-best-model-at-end/44188>\
**Category:** 🤗Transformers\
**Created:** [June 22, 2023, 12:55pm UTC](https://discuss.huggingface.co/t/hyperparameter-optimization-and-load-best-model-at-end/44188 "2023-06-22T12:55:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ejschwartz](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ejschwartz/32/16902_2.png) [@ejschwartz](https://discuss.huggingface.co/u/ejschwartz)\
**Post date:** [June 22, 2023, 12:55pm UTC](https://discuss.huggingface.co/t/hyperparameter-optimization-and-load-best-model-at-end/44188/1 "2023-06-22T12:55:45Z")

</div>

When optimizing hyperparameters, in the scope of a particular trial, is there a way to return the “best” model as in the `load_best_model_at_end` option? In other words, rather than optimizing the final metric of the trial, I’d like to optimize the best value of the metric observed in the trial, even if that was not at the end.

To be very specific, let’s say my accuracy metric looks like this for a trial:

0.5, 0.8. 0.9, 0.85

I’d like to return the model with 0.9 accuracy to the optimizer, not 0.85. (This is particularly useful with early stopping)

I’m currently using the wandb backend, and although I have `load_best_model_at_end` enabled, wandb is always seeing the most recent model.

---

<div class="post-metadata">

**Author:** ![rvienne](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rvienne](https://discuss.huggingface.co/u/rvienne)\
**Post date:** [June 22, 2023, 5:51pm UTC](https://discuss.huggingface.co/t/hyperparameter-optimization-and-load-best-model-at-end/44188/2 "2023-06-22T17:51:58Z")

</div>

Hello, @ejschwartz.  
My go-to in this kind of situation is to add a “best\_metric” key to the eval dictionary output.  
This metric should be updated at each eval as the max between the current value and the new one to keep memory of the best metric obtained during the trial.

Let’s take an example:  
If I’m doing a multiclass classification and I want to select my best model as the one which maximizes micro\_avg\_f1-score for each trial, among all values obtained during the trial.

I’ll modify my compute\_metrics function to add a “best\_metric” key, computed as the max between the current value (initialized at 0.0) and the current eval value for my metric of choice (here micro\_avg\_f1-score).

Then my compute\_objective function should be as simple as:  
return metrics[“eval\_best\_metric”]

Does that help you ?

---

<div class="post-metadata">

**Author:** ![ejschwartz](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ejschwartz/32/16902_2.png) [@ejschwartz](https://discuss.huggingface.co/u/ejschwartz)\
**Post date:** [June 22, 2023, 6:13pm UTC](https://discuss.huggingface.co/t/hyperparameter-optimization-and-load-best-model-at-end/44188/3 "2023-06-22T18:13:19Z")

</div>

Thanks, this seems like it should work. I’ll give it a try.