# Looking for hyperparameter tuning advices

**URL:** <https://discuss.huggingface.co/t/looking-for-hyperparameter-tuning-advices/25473>\
**Category:** Beginners\
**Created:** [November 3, 2022, 12:11pm UTC](https://discuss.huggingface.co/t/looking-for-hyperparameter-tuning-advices/25473 "2022-11-03T12:11:27Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jorgenhw](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jorgenhw/32/5005_2.png) [@jorgenhw](https://discuss.huggingface.co/u/jorgenhw)\
**Post date:** [November 3, 2022, 12:11pm UTC](https://discuss.huggingface.co/t/looking-for-hyperparameter-tuning-advices/25473/1 "2022-11-03T12:11:27Z")

</div>

I’m finetuning a language model (vesteinn/ScandiBERT) on a multiclass text classification (pos, neg, neu) task and I wish to run a hyperparameter search to find the optimal hyperparameters. As I’m starting to read up on it I realize that HP tuning is a wild west with no consensus on which HP to tune on and which ranges to specify for these. I’m not an NLP expert so I might be missing some conceptual knowledge which is why I’m hoping that one of you would take a look at how I initiate my HP tuning using Optuna:  
I looks like this:

```auto
!pip install -q optuna

def model_init():
    return model #model = AutoModelForSequenceClassification.from_pretrained(model, num_labels=3) (defined earlier)

training_args = TrainingArguments(output_dir=model_name, 
                                  evaluation_strategy = "epoch",
                                  save_strategy = "epoch", 
                                  num_train_epochs = epochs, 
                                  per_device_train_batch_size = batch_size,
                                  per_device_eval_batch_size = batch_size,
                                  learning_rate = learning_rate,
                                  weight_decay=0.01,
                                  load_best_model_at_end=True,
                                  report_to="wandb")

trainer = Trainer(
    model_init=model_init,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["test"],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)

def my_hp_space(trial):
    return {
        "learning_rate": trial.suggest_float("learning_rate", 1e-6, 1e-4, log=True),
        "num_train_epochs": trial.suggest_int("num_train_epochs", 5, 15),
        #"seed": trial.suggest_int("seed", 1, 40),
        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [4, 8, 16, 32, 64]),
        "weight_decay": trial.suggest_loguniform('weight_decay', 1e-4, 1e-2)
    }

import optuna
from optuna.samplers import TPESampler
sampler = optuna.samplers.TPESampler()
pruner = optuna.pruners.SuccessiveHalvingPruner()

best_run = trainer.hyperparameter_search(
    n_trials=100, 
    direction="maximize", 
    hp_space=my_hp_space, 
    backend = "optuna",
    sampler = sampler,
    pruner = pruner
    )

for n, v in best_run.hyperparameters.items():
    setattr(trainer.args, n, v) # for running the experiment with the best hyperparameters from the hyperparameters search

trainer.train() # argument trial can be used for hyperparameter search

```

My code does run but I’m uncertain of whether I’m missing anything important in my value definitions. Especially since the trial gets pruned from run 3 - 99. Here is a link to my notebook with all the code.[Google Colab](https://colab.research.google.com/drive/1Sluei065T_VH_XgXQi85OzmfPBlVoARc?usp=sharing)