# Transformers and Hyperparameter search using Optuna

**URL:** <https://discuss.huggingface.co/t/transformers-and-hyperparameter-search-using-optuna/38429>\
**Category:** 🤗Transformers\
**Created:** [May 2, 2023, 11:37am UTC](https://discuss.huggingface.co/t/transformers-and-hyperparameter-search-using-optuna/38429 "2023-05-02T11:37:46Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Petrina](https://avatars.discourse-cdn.com/v4/letter/p/a3d4f5/32.png) [@Petrina](https://discuss.huggingface.co/u/Petrina)\
**Post date:** [May 2, 2023, 11:37am UTC](https://discuss.huggingface.co/t/transformers-and-hyperparameter-search-using-optuna/38429/1 "2023-05-02T11:37:46Z")

</div>

Hello all,

I am currently trying to perform hyper-parameter search using trainer.hyperparameter\_search(), using optuna as a back end. However, I have some questions regarding the several trials that are performed:

1. Is there a way to retrieve the whole study object (with all trials and all metrics) and not only the best trial?

2. Is the pruning of trials activated? I can see in the source code the relevant command (report of some intermediate metric and then the check trial.should\_prune(): ), however I am not sure that is activated by the hyperparameter\_search(). Ideally, I would like to have a report of the trials pruned and the ones that finished.

Thank you in advance,  
Petrina

---

<div class="post-metadata">

**Author:** ![The-Fanta](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/the-fanta/32/15008_2.png) [@The-Fanta](https://discuss.huggingface.co/u/The-Fanta)\
**Post date:** [May 3, 2023, 2:03pm UTC](https://discuss.huggingface.co/t/transformers-and-hyperparameter-search-using-optuna/38429/2 "2023-05-03T14:03:57Z")

</div>

1. Not to my knowledge. But you can retrieve a log of all the Optuna trials (Hugging Face calls them “runs”), in two different ways.

_First way_  
If you check the `Trainer` object after you have called its `hyperparameter_search()` method, you will find the log in `Trainer.state.log_history`, it’s a list.

_Second way_  
Also, the logs are saved on disk as JSON files called `trainer_state.json` every time a checkpoint is saved during the optimization process. You can set the directory to store checkpoints with the `output_dir` parameter of `TrainingArguments()`; I recommend you also set its `logging_strategy` and `save_strategy` to `'epoch'` such that you get one checkpoint, and therefore one log saved, at the end of every epoch.

BTW, if you find a way to fetch the whole `Study` object, please do let me know.

In addition to the above, you can get very detailed information about what Optuna is doing by enabling its persistent storage on a SQLite database. It is straightforward. To browse the database info I then use DBeaver CE. To enable the database storage, you can pass parameters `storage` and `load_if_exists` to `Trainer.hyperparameter_search()`, e.g.:

```auto
res = trainer.hyperparameter_search(hp_space=hp_space,
    n_trials=params.fine_tuning.n_trials,
    direction='maximize',
    compute_objective=compute_objective,
    sampler=optuna_sampler,
    study_name=study_name,
    storage='sqlite:///my_optuna_studies.db',
    load_if_exists=True,
    pruner=NopPruner())

```

Those parameters are then forwarded by the `Trainer` to `optuna.create_study()`, see the [Optuna documentation](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html?highlight=create_study#optuna.study.create_study) to see their usage.

1. Yes, by default, unless you disable it. If you enable persistence in the database with Optuna (as per point above) you can find in the database information about which trials have been pruned, in the `trials` table, `state` column.

To disable pruning, you can pass parameter `pruner=NopPruner()` to `Trainer.hyperparameter_search()`, like I did in the snippet of code at point 1. See Optuna documentation to chose the pruning strategy instead.

There are a few caveats I have stumbled upon, I am still investigating but they may interest you.  
a) after hyperparameters search, `Trainer.state.best_model_checkpoint` doesn’t contain the path to the _best_ checkpoint saved, but to the _last_ checkpoint saved instead. If I am getting this right, it is a Hugging Face bug.  
b) Optuna’s DB persistence should allow you to interrupt and then resume the hyperparameters search. While I have that feature work correctly when using just Optuna, in a toy example, so far I couldn’t make it work with Hugging Face: when I try to resume the hyperparameters search, it restarts from the beginning instead, ignoring trials from the previous experiment. Perhaps I am doing something wrong here.

---

<div class="post-metadata">

**Author:** ![Petrina](https://avatars.discourse-cdn.com/v4/letter/p/a3d4f5/32.png) [@Petrina](https://discuss.huggingface.co/u/Petrina)\
**Post date:** [May 4, 2023, 12:48pm UTC](https://discuss.huggingface.co/t/transformers-and-hyperparameter-search-using-optuna/38429/3 "2023-05-04T12:48:38Z")

</div>

Thank you very much for your analytical answer!

For point 1, I have also noticed 2 things that are not straight-forward to me, but yet I haven’t put much effort into them. Just mentioning them in case you have seen something similar:

My code for hyper parameter search is the following:

```auto
  training_args = TrainingArguments(         
      output_dir=SAVE_DIR,      
      per_device_train_batch_size=PER_DEVICE_TRAIN_BATCH,         
      per_device_eval_batch_size=PER_DEVICE_EVAL_BATCH,
      evaluation_strategy = "epoch",
      logging_strategy="epoch")     
    
  trainer = Trainer(
    model=None,
    args=training_args,
    train_dataset=tokenized_dset['train'],
    eval_dataset=tokenized_dset['validation'],
    model_init=model_init
  )
    
  best_trial = trainer.hyperparameter_search(
  direction="minimize",
  backend="optuna",
  hp_space=optuna_hp_space,
  n_trials=NUM_TRIALS
  )

```

1. Even though I have set the `logging_strategy="epoch"`, I get logs every 500 steps, so in the directory I have checkpoint-500, checkpoint-1000 etc., which is confusing since every log contains several epochs.

2. In the directory there is a json file that contains the whole history of epochs (It could be the same as trainer.state.log\_history), however the metrics of the last epoch aren’t logged in this file, if I have noticed correctly.

For point 2, yes I run a study with 10 trials and I noticed that some where pruned, so I guess it is working. I will definitely consider the SQLite database since it seems very handy in keeping all the information about trials.

Thank you for the caveats too. I will keep them in mind and in case I find a way to have as an object the whole study, I will let you know!

Thanks again,  
Petrina

---

<div class="post-metadata">

**Author:** ![The-Fanta](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/the-fanta/32/15008_2.png) [@The-Fanta](https://discuss.huggingface.co/u/The-Fanta)\
**Post date:** [May 4, 2023, 1:04pm UTC](https://discuss.huggingface.co/t/transformers-and-hyperparameter-search-using-optuna/38429/4 "2023-05-04T13:04:33Z")

</div>

> [@Petrina](#):
>
> Even though I have set the `logging_strategy="epoch"`, I get logs every 500 steps, so in the directory I have checkpoint-500, checkpoint-1000 etc., which is confusing since every log contains several epochs.

Steps with transformers are not epochs. In my understanding, 1 step = 1 batch  
So if you have a train set with, say, 1000 samples, and you train in batches of 64 samples, then it will take you 16 steps (16 batches) to push the whole train set through training once, i.e. to do 1 epoch. In this example, 1 epoch = 16 steps. If you do the math for your script, you should find that, in your case, 500 steps = 1 epoch.

> [@Petrina](#):
>
> In the directory there is a json file that contains the whole history of epochs (It could be the same as trainer.state.log\_history), however the metrics of the last epoch aren’t logged in this file, if I have noticed correctly.

Not sure I understand which .json file you are referring to. What is the complete path?  
Log fiiles I have are organized like this

 ![Screenshot from 2023-05-04 14-59-56](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/76f2a0607f6ceb5ed68e294703351d6e3f36056e.png)

There is one directory for every run (what Optuna calls “trials”), and every run contains its checkpoints, in my case they are saved every 32 steps because here I have 1 epoch = 32 steps . In every checkpoint, `trainer_state.json` contains all the info for that specific run up to the given checkpoint included.

If you are looking for a single, overall log file with all the runs together, no I don’t have it. I get one log file `trainer_state.json` for every checkpoint of every run.

---

<div class="post-metadata">

**Author:** ![Petrina](https://avatars.discourse-cdn.com/v4/letter/p/a3d4f5/32.png) [@Petrina](https://discuss.huggingface.co/u/Petrina)\
**Post date:** [May 5, 2023, 7:33am UTC](https://discuss.huggingface.co/t/transformers-and-hyperparameter-search-using-optuna/38429/5 "2023-05-05T07:33:34Z")

</div>

> [@The-Fanta](#):
>
> Steps with transformers are not epochs. In my understanding, 1 step = 1 batch  
> So if you have a train set with, say, 1000 samples, and you train in batches of 64 samples, then it will take you 16 steps (16 batches) to push the whole train set through training once, i.e. to do 1 epoch. In this example, 1 epoch = 16 steps. If you do the math for your script, you should find that, in your case, 500 steps = 1 epoch.

Ok yes, this clarifies the issue. The dataset I had would result in 1 epoch=500 steps, and because the default number of steps to log when `logging_strategy="steps"` was 500, I thought that this logs anyways every 500 steps. Thank you very much for this!

> [@The-Fanta](#):
>
> Not sure I understand which .json file you are referring to. What is the complete path?  
> Log fiiles I have are organized like this

Yes, you understood correctly, thank you for the clarifications on this too.

Also, for the trainer.state.log\_history, just to underline that it contains only the log of the last trial. So, as you stated, I understand there is not an obvious way to get the logs for the wholes study.

Thank you very much once again for the help!