# Finetuning longformer

**URL:** <https://discuss.huggingface.co/t/finetuning-longformer/15732>\
**Category:** Models\
**Created:** [March 15, 2022, 4:47pm UTC](https://discuss.huggingface.co/t/finetuning-longformer/15732 "2022-03-15T16:47:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![nitempe](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nitempe](https://discuss.huggingface.co/u/nitempe)\
**Post date:** [March 15, 2022, 4:47pm UTC](https://discuss.huggingface.co/t/finetuning-longformer/15732/1 "2022-03-15T16:47:15Z")

</div>

I am building a longformer based classification model similar to [this](https://github.com/jlealtru/website_tutorials/blob/main/notebooks/Longformer%20with%20IMDB.ipynb). If I want to tune my model, which parameters do I need to consider and are there any recommendations for them.

currently I am thinking about below parameters and their values as below

`attention_window`=256, 512, or 1024  
`optim`=“adamw\_torch, adamw\_apex\_fused, or adafactor”  
`weight_decay`=0,0.01,0.02  
`learning_rate`=5e-5,10e-5

what other parameters should I tune? Do I need to tune any of these?  
`num_train_epochs`, `per_device_train_batch_size`, `gradient_accumulation_steps`, `per_device_eval_batch_size`, `warmup_steps`, `dataloader_num_workers`, `lr_scheduler_type`,

Please let me know if there is any documentation about parameter tuning

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [March 16, 2022, 8:26am UTC](https://discuss.huggingface.co/t/finetuning-longformer/15732/2 "2022-03-16T08:26:34Z")

</div>

Hi,

The Trainer supports hyperparameter search using [Optuna](https://optuna.org/) or [Ray Tune](https://docs.ray.io/en/latest/tune/). Check the bottom of the [official text classification notebook](https://github.com/huggingface/notebooks/blob/master/examples/text_classification.ipynb) for more info.

---

<div class="post-metadata">

**Author:** ![nitempe](https://avatars.discourse-cdn.com/v4/letter/n/54ee81/32.png) [@nitempe](https://discuss.huggingface.co/u/nitempe)\
**Post date:** [March 18, 2022, 12:40am UTC](https://discuss.huggingface.co/t/finetuning-longformer/15732/3 "2022-03-18T00:40:52Z")

</div>

it seems that only parameters that are being tuned are below. Is there any reason to only tune those? should i look into tuning others?  
`hyperparameters={'learning_rate': 4.357724525964853e-05, 'num_train_epochs': 2, 'seed': 38, 'per_device_train_batch_size': 32})`