# T5 Finetuning Tips

**URL:** <https://discuss.huggingface.co/t/t5-finetuning-tips/684>\
**Category:** Models\
**Created:** [August 11, 2020, 8:39pm UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684 "2020-08-11T20:39:06Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![moscow25](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moscow25/32/259_2.png) [@moscow25](https://discuss.huggingface.co/u/moscow25)\
**Post date:** [August 12, 2020, 6:08am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/3 "2020-08-12T06:08:06Z")

</div>

- Training with AdaFactor works quite well for me so far. I use the “constant LR 0.001” recommended in all of Colin Raffel’s finetuning paper and other AdaFactor settings from original Noam Shazeer paper.
- Fair-Seq’s AdaFactor implementation is good – except you need to turn auto-scaling options off – no idea why they are on by default in the init.
- [https://github.com/pytorch/fairseq/blob/775122950d145382146e9120308432a9faf9a9b8/fairseq/optim/adafactor.py](https://github.com/pytorch/fairseq/blob/775122950d145382146e9120308432a9faf9a9b8/fairseq/optim/adafactor.py)
- lr=0.001, scale\_parameter=False, relative\_step=False

This even “works” in FP16 – but don’t get me started on Native AMP quite yet…

But in summary – I would strongly recommend using AdaFactor and not ADAM for T5 training and finetuning.

- this is what the T5 authors use themselves
- AdaFactor was developed specifically with Transformers/T5 in mind (say so in the paper)
- ADAM is a massive waste of memory in general; it’s not surprising that something more efficient would work as well unless you have custom additions to your model

---

_[View the full topic](https://discuss.huggingface.co/t/t5-finetuning-tips/684)._