# Using hyperparameter-search in Trainer

**URL:** <https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785>\
**Category:** 🤗Transformers\
**Created:** [August 20, 2020, 12:58pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785 "2020-08-20T12:58:34Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 20, 2020, 12:58pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/1 "2020-08-20T12:58:34Z")

</div>

This [branch](https://github.com/huggingface/transformers/commit/168d42b9600ab5a1ea125448560e186f83109956) hasn’t been merged, but I want to use optuna in my workflow. Although I have tried it, I want to confirm the usage. @sgugger (firstly thanks for the PR) could you please provide instructions on what changes do I need to make to make it work (like defining the search space and then getting results on them, and finding the best hyperparams). I want to confirm if I’m using it in the right manner. Also is the implementation complete ?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 20, 2020, 1:01pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/2 "2020-08-20T13:01:43Z")

</div>

Hi there!

This is a work in progress so I’d hold on a tiny bit before starting using it (I’ll actually make some changes today). I’ll add an example in the PR once I’m done (hopefully by end of day) so you (and others) can start playing with it and give us potential feedback, but be prepared for some slight changes in the API as we polish it (we want to support other hp-search platforms such as Ray)

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 20, 2020, 3:54pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/3 "2020-08-20T15:54:14Z")

</div>

Thanks for the reply. I’ll look forward to the example and using it. I’ll hopefully try to contribute if I come across some rough edges. `Trainer` changes a lot, my inherited trainer code breaks most of the time after each update, so I’m prepared for it 😉.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 20, 2020, 8:42pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/4 "2020-08-20T20:42:47Z")

</div>

Ok, done for today and prepared the road to support ray as well (not working right now though). There is an example on a regression problem in the README cause I didn’t want to launch my GPU setup. Will add a real example soon, but it should be enough to get you going.

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 21, 2020, 5:12am UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/5 "2020-08-21T05:12:31Z")

</div>

Could you please tell where that README is ? I checked your recent commits on both `trainer_optuna` branch and `master`, didn’t see it.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 21, 2020, 1:11pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/6 "2020-08-21T13:11:12Z")

</div>

Sorry, not README, I meant the [PR](https://github.com/huggingface/transformers/pull/6576) first post.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 21, 2020, 5:32pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/7 "2020-08-21T17:32:45Z")

</div>

I put a real example now.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [August 21, 2020, 5:37pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/8 "2020-08-21T17:37:06Z")

</div>

What are the pros/cons of optuna VS ray?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 21, 2020, 5:40pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/9 "2020-08-21T17:40:14Z")

</div>

Both work with the API. I haven’t used either long enough to have a strong opinion, but basically ray would be better if you have multiple GPUs and optuna might be better with just one, from what I understood.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 24, 2020, 4:02pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/10 "2020-08-24T16:02:12Z")

</div>

FYI, this has been merged in master. Here is an example of use:

```auto
from nlp import load_dataset, load_metric
from transformers import AutoModelForSequenceClassification, AutoTokenizer, DataCollatorWithPadding, Trainer, TrainingArguments

tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
dataset = load_dataset('glue', 'mrpc')
metric = load_metric('glue', 'mrpc')

def encode(examples):
    outputs = tokenizer(examples['sentence1'], examples['sentence2'], truncation=True)
    return outputs

encoded_dataset = dataset.map(encode, batched=True)
# Won't be necessary when this PR is merged with master since the Trainer will do it automatically
encoded_dataset.set_format(columns=['attention_mask', 'input_ids', 'token_type_ids', 'label'])

def model_init():
    return AutoModelForSequenceClassification.from_pretrained('bert-base-cased', return_dict=True)

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    predictions = predictions.argmax(axis=-1)
    return metric.compute(predictions=predictions, references=labels)

# Evaluate during training and a bit more often than the default to be able to prune bad trials early.
# Disabling tqdm is a matter of preference.
training_args = TrainingArguments("test", evaluate_during_training=True, eval_steps=500, disable_tqdm=True)
trainer = Trainer(
    args=training_args,
    data_collator=DataCollatorWithPadding(tokenizer),
    train_dataset=encoded_dataset["train"], 
    eval_dataset=encoded_dataset["validation"], 
    model_init=model_init,
    compute_metrics=compute_metrics,
)

# Defaut objective is the sum of all metrics when metrics are provided, so we have to maximize it.
trainer.hyperparameter_search(direction="maximize")

```

This will use optuna or Ray Tune, depending on which you have installed. If you have both, it will use optuna by default, but you can pass `backend="ray"` to use Ray Tune. Note that you need an installation from source of nlp to make the example work.

To customize the hyperparameter search space, you can pass a function `hp_space` to this call. Here is an example if you want to search higher learning rates than the default with optuna:

```auto
def my_hp_space(trial):
    return {
        "learning_rate": trial.suggest_float("learning_rate", 1e-4, 1e-2, log=True),
        "num_train_epochs": trial.suggest_int("num_train_epochs", 1, 5),
        "seed": trial.suggest_int("seed", 1, 40),
        "per_device_train_batch_size": trial.suggest_categorical("per_device_train_batch_size", [4, 8, 16, 32, 64]),
    }

trainer.hyperparameter_search(direction="maximize", hp_space=my_hp_space)

```

and ray:

```auto
def my_hp_space_ray(trial):
    from ray import tune

    return {
        "learning_rate": tune.loguniform(1e-4, 1e-2),
        "num_train_epochs": tune.choice(range(1, 6)),
        "seed": tune.choice(range(1, 41)),
        "per_device_train_batch_size": tune.choice([4, 8, 16, 32, 64]),
    }

trainer.hyperparameter_search(direction="maximize", hp_space=my_hp_space)

```

If you want to customize the objective to minimize/maximize, pass along a function to `compute_objective`:

```auto
def my_objective(metrics):
    # Your elaborate computation here
    return result_to_optimize

trainer.hyperparameter_search(direction="maximize", compute_objective=my_objective)

```

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [August 24, 2020, 4:23pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/11 "2020-08-24T16:23:56Z")

</div>

Thanks. I was following this PR. I wanted to know which type of hyperparams can be tuned with this approach? Does it work with Default ones only (training\_args) ? What if we have custom param that we want to tune (for instance a lambda in an objective function) ?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 24, 2020, 5:18pm UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/12 "2020-08-24T17:18:51Z")

</div>

The hyperparams you can tune must be in the `TrainingArguments` you passed to your Trainer. If you have custom ones that are not in `TrainingArguments`, just subclass `TrainingArguments` and add them in your subclass.

The `hp_space` function indicates the hyperparameter search space (see the code of the default for optuna or Ray in `training_utils.py` and adapt it to your needs) and the `compute_objective` function should return the objective to minize/maximize.

---

<div class="post-metadata">

**Author:** ![aswincandra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aswincandra/32/305_2.png) [@aswincandra](https://discuss.huggingface.co/u/aswincandra)\
**Post date:** [August 31, 2020, 8:48am UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/14 "2020-08-31T08:48:30Z")

</div>

Thank you so much! But I have a problem when defining the Trainer. It said, “ **init** () got an unexpected keyword argument ‘model\_init’”. Is the Trainer doesn’t recognize the ‘model\_init’ argument?

I think this error affect next error when I want to call the ‘hyperparameter\_search’ method. It said, “‘Trainer’ object has no attribute ‘hyperparameter\_search’”.

What should I do? Very sorry for the very newbie question 🙏 and Thankyou before.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [August 31, 2020, 11:38am UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/15 "2020-08-31T11:38:14Z")

</div>

This is new so you need an installation from source to use it. It will be in the next release coming soon otherwise.

---

<div class="post-metadata">

**Author:** ![aswincandra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aswincandra/32/305_2.png) [@aswincandra](https://discuss.huggingface.co/u/aswincandra)\
**Post date:** [September 3, 2020, 1:16am UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/16 "2020-09-03T01:16:03Z")

</div>

Alright, I’m waiting for it! 🚀

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [September 3, 2020, 6:26am UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/17 "2020-09-03T06:26:43Z")

</div>

FYI, You can pip install now to use this feature. No need to build from source.

---

<div class="post-metadata">

**Author:** ![aswincandra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aswincandra/32/305_2.png) [@aswincandra](https://discuss.huggingface.co/u/aswincandra)\
**Post date:** [September 10, 2020, 3:35am UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/18 "2020-09-10T03:35:00Z")

</div>

Oh yeah thank you, It seems developed. But I’m still getting problem in `hyperparameter_search` method. I defined my backend parameter to ‘optuna’ but the error said: `You picked the optuna backend, but it is not installed. Use pip install optuna.`, though I’ve already pip-installed it before the `hyperparameter_search` code line. The case was same when I defined the backend parameter into ‘ray’. Have I make a mistake? I run my code in Google Colab by the way.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [September 10, 2020, 11:33am UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/19 "2020-09-10T11:33:52Z")

</div>

It means that it is not installed in your current environment. If you are using notebooks, you have to restart the kernel. Python needs to reload the libraries to see which ones are available.

---

<div class="post-metadata">

**Author:** ![aswincandra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aswincandra/32/305_2.png) [@aswincandra](https://discuss.huggingface.co/u/aswincandra)\
**Post date:** [September 14, 2020, 4:34am UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/20 "2020-09-14T04:34:35Z")

</div>

Oh yeah it has already worked. Thank you so much! 😁

---

<div class="post-metadata">

**Author:** ![chinhon](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chinhon/32/2966_2.png) [@chinhon](https://discuss.huggingface.co/u/chinhon)\
**Post date:** [September 14, 2020, 6:06am UTC](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/21 "2020-09-14T06:06:42Z")

</div>

I wonder if Sylvain or others might have advice on how to make the hyperparameters search more efficient or manageable, time and resource-wise.

I’ve tried slimming down the dataset (500K rows to 90K rows), reducing the number of parameters to tune (to just 1, number of epochs) and changing the “direction” to “minimize” instead of “maximize”.

Is there something else I can do, aside from further cutting down the size of the dataset? I’m running trials on Colab Pro with GPU/high-RAM enabled, and current version looks like it’ll take about 7 hours (perfectly fine for others I’m sure).

I don’t suppose there’s an equivalent of RandomizedSearchCV for trainer?

[Next page](https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785.md?page=2)