# Hyper params search for model config

**URL:** <https://discuss.huggingface.co/t/hyper-params-search-for-model-config/74259>\
**Category:** 🤗Transformers\
**Created:** [February 22, 2024, 4:52am UTC](https://discuss.huggingface.co/t/hyper-params-search-for-model-config/74259 "2024-02-22T04:52:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nevermetyou](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nevermetyou/32/21618_2.png) [@Nevermetyou](https://discuss.huggingface.co/u/Nevermetyou)\
**Post date:** [February 22, 2024, 4:52am UTC](https://discuss.huggingface.co/t/hyper-params-search-for-model-config/74259/1 "2024-02-22T04:52:44Z")

</div>

Hello everyone

I am currently training the t5 model for the seq2seq task right now, and I wonder if we can use Optuna to do the hyper-params search for the config like `num_layers` OR `num_heads`.

According to Mr. Sgugger in this discussion

> The hyperparams you can tune must be in the `TrainingArguments` you passed to your Trainer. If you have custom ones that are not in `TrainingArguments`, just subclass `TrainingArguments` and add them in your subclass.
> 
> The `hp_space` function indicates the hyperparameter search space (see the code of the default for optuna or Ray in `training_utils.py` and adapt it to your needs) and the `compute_objective` function should return the objective to minize/maximize.

But it was almost 3 years ago, I am wondering there is any method to achieve this now??  
Because `num_layers` or `num_heads` are not in the args of Trainer, I really don’t know how to search for this.