# T5 Finetuning Tips **URL:** \ **Category:** Models\ **Created:** [August 11, 2020, 8:39pm UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684 "2020-08-11T20:39:06Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 11, 2020, 8:39pm UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/1 "2020-08-11T20:39:06Z")
Starting this for results, sharing + tips and tricks, and results. This is my first attempt at this kind of thread so it may completely fail. Some things I’ve found - Apparently if you copy AdaFactor from fairseq, as recommended by t5 authors, you can fit batch size = 2 for t5-large lm finetuning - fp16 rarely works. - for most tasks, you need to manually add `` to the end of your sequence. Thing’s I’ve read - task specific prefix doesn’t matter much. cc @mrm8488 @valhalla @patrickvonplaten who have all tried different experiments. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [August 12, 2020, 5:36am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/2 "2020-08-12T05:36:22Z")
Things I’ve found - task prefixes matter when 1. When doing multi-task training 2. When your task similar or related to one of the supervised tasks used in T5 pre-training mixture. - Needs slightly higher LR than the default one set in Trainer, in my experiments 1e-4 and 3e-4 worked for almost all problems (classification, QA, que-gen, summ) - no need to pass `decoder_input_ids` to T5 yourself, just pass `labels` and the `T5Model` will prepare them for you. labels should end with `eos_token`. (important! This is where most of the mistakes are happening). - T5 uses `pad_token` as the `decoder_start_token_id` so when doing generation without the `generate` function make sure you start it with pad token. - trimming batches when training on TPU leads to very slower training. - apparently, because of sentencepiece and some possible leakage of other languages in C4 data, T5 gives somewhat sensible results for french lang. fine-tuned it on FQuAD (french version of SQuAD) for que gen and BLUE-4 against dev set was 15. Not sure if it’s an issue or not but in some cases using `label_smoothing` in T5 resulted in `nan` loss ---
**Author:** ![moscow25](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moscow25/32/259_2.png) [@moscow25](https://discuss.huggingface.co/u/moscow25)\ **Post date:** [August 12, 2020, 6:08am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/3 "2020-08-12T06:08:06Z")
- Training with AdaFactor works quite well for me so far. I use the “constant LR 0.001” recommended in all of Colin Raffel’s finetuning paper and other AdaFactor settings from original Noam Shazeer paper. - Fair-Seq’s AdaFactor implementation is good – except you need to turn auto-scaling options off – no idea why they are on by default in the init. - [https://github.com/pytorch/fairseq/blob/775122950d145382146e9120308432a9faf9a9b8/fairseq/optim/adafactor.py](https://github.com/pytorch/fairseq/blob/775122950d145382146e9120308432a9faf9a9b8/fairseq/optim/adafactor.py) - lr=0.001, scale\_parameter=False, relative\_step=False This even “works” in FP16 – but don’t get me started on Native AMP quite yet… But in summary – I would strongly recommend using AdaFactor and not ADAM for T5 training and finetuning. - this is what the T5 authors use themselves - AdaFactor was developed specifically with Transformers/T5 in mind (say so in the paper) - ADAM is a massive waste of memory in general; it’s not surprising that something more efficient would work as well unless you have custom additions to your model ---
**Author:** ![moscow25](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moscow25/32/259_2.png) [@moscow25](https://discuss.huggingface.co/u/moscow25)\ **Post date:** [August 12, 2020, 6:09am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/4 "2020-08-12T06:09:53Z")
those observations seem quite consistent with out experience as well – did not try on TPU yet ---
**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](https://discuss.huggingface.co/u/Jung)\ **Post date:** [August 12, 2020, 8:03am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/5 "2020-08-12T08:03:04Z")
Hi guys! I just finish training T5-large on ELI5 on 270,000 exampels using TPU V2-8 on colab modified from @valhalla notebook! This is not really finetuning tips, but some tips to make T5-large trainable on TPU V2-8 . T5-large is challenging to train on TPU V2-8 with Pytorch (for me) - I faced a lot of memory problem (even on Colab High-RAM instance), this [notebook of Davide Libenzi - one of XLA authors](https://www.kaggle.com/davidelibenzi/simple-xlmr-tpu-pytorch) suggested to declare large model outside `_mp_fn` (see his `mx` variable ) - with T5-base , there is around 7 minutes overhead before training can start, for T5-large, this takes 1 hour overhead to me - with `max_length = 128` (both input and target), I am able to set `per_device_train_batch_size = 4` (so, global\_batch\_size = 4\*8 = 32) - there is an issue that `xm.save()` causes memory error with large models like XLM-Roberta , it happen to T5-large too, so I have to ignore the default `save_steps` of `Trainer` by setting it to 1000000 Combine all these, took me around 1 day before I can make a trainable notebook, so hopefully these tricks can be useful to some of you guys too! I would like to find time to make a TF2 version which should be more stable on TPU 🙂 More note - As @valhalla mentioned in his notebook, High-RAM instance is a must. Lately Kaggle notebook increased RAM to 16GB for TPUV3-8, but I could not the training to success (sadly since V3-8 should be 2x faster than V2-8) ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [August 12, 2020, 2:05pm UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/6 "2020-08-12T14:05:51Z")
Hi @moscow25, thanks for sharing the AdaFactor info. I was wondering if constant LR of 1e-3 is working for small batch sizes, because in the paper they mentioned that the BS for fine-tuning was 128, it’s not possible to use 128 BS with single V100 for model \>t5-base . ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [August 12, 2020, 2:10pm UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/7 "2020-08-12T14:10:34Z")
This trick of loading the model outside of \_map\_fn is awesome! It should save some memory. In pytorch-xla the model and the datset is loaded in all processes (8 in case 8 TPU cores) so it ends up taking lot of memory. Lazy loading dataset should also reduce RAM usage. On V3-8, I was able to use bs of 8 per device with max\_source\_length 512 and max\_target\_length 64 ---
**Author:** ![moscow25](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moscow25/32/259_2.png) [@moscow25](https://discuss.huggingface.co/u/moscow25)\ **Post date:** [August 12, 2020, 4:56pm UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/8 "2020-08-12T16:56:34Z")
Sure thing @valhalla. I did not try too many settings… but LR 0.001 seems to work just fine for smaller finetuning batches. I’m running global batch of 2\*8 [2 per GPU] – though with a bit of gradient accumulation (4x I believe) but tbh it’s not really that sensitive as far as I can tell. The only gotcha is to turn off those extra scaling parameters that FAIR-seq threw in there and set True by default for no good reason. (scale\_parameter=False, relative\_step=False) To get bigger batches, I’m pretty sure we need to add some gradient checkpointing to the model. Trying that out next… ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 19, 2020, 4:20am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/9 "2020-08-19T04:20:04Z")
T5 questions I think I know the answer to that multiple people have asked. Correct me if I’m wrong! Quotes are from the paper. Q: What masking objective did they use for pretraining? Span Corruption. > Specifically, we use a mean span length of 3 and corrupt 15% of the original sequence. We found that this objective produced marginally better performance (Table 7) while being slightly more computationally efficient due to shorter target sequence lengths. Q: Are the hf checkpoints trained with multi-tasking? A: yes Q:Do we have access to T5 1.1 Checkpoints: A: No, because they are not obvious wins: [Should I use t5v1.1, t5narrow and TalkingHeads? · Issue #266 · google-research/text-to-text-transfer-transformer · GitHub](https://github.com/google-research/text-to-text-transfer-transformer/issues/266) ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [August 19, 2020, 9:00am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/10 "2020-08-19T09:00:10Z")
More on T5 pre-training objective Each corrupted span is replaced by a unique sentinel token. . The output sequence then consists of the dropped-out spans, delimited by the sentinel tokens used to replace them in the input plus a final sentinel token. T5 uses 100 extra ids as sentinel tokens (` ... `) from HF docs > output sequence is formed as a concatenation of the same sentinel tokens and the _real_ masked tokens _E.g._ the sentence “The cute dog walks in the park” with the masks put on “cute dog” and “the” should be processed as follows: ```python3 input_text = "The walks in park" target_text = " cute dog the " ``` ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 20, 2020, 1:38am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/11 "2020-08-20T01:38:46Z")
Has anyone managed to finetune t5 in fp16? Maybe @mrm8488 ? If so which torch version/how ? ---
**Author:** ![mrm8488](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mrm8488/32/5836_2.png) [@mrm8488](https://discuss.huggingface.co/u/mrm8488)\ **Post date:** [August 20, 2020, 4:22am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/12 "2020-08-20T04:22:52Z")
No, I didn’t. But I will check it out!! ---
**Author:** ![saareliad](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@saareliad](https://discuss.huggingface.co/u/saareliad)\ **Post date:** [August 21, 2020, 10:42am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/13 "2020-08-21T10:42:33Z")
Hi, what about packed sequences they use in paper? I’m trying to figure out how to do it in huggingface model (to replicate an experiment) ---
**Author:** ![saareliad](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@saareliad](https://discuss.huggingface.co/u/saareliad)\ **Post date:** [August 21, 2020, 10:56am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/14 "2020-08-21T10:56:30Z")
I see that in T5 they do use scale\_parameter [https://console.cloud.google.com/storage/browser/\_details/t5-data/experiments/scaling/sc-bi\_v1-bsx4/operative\_config.gin](https://console.cloud.google.com/storage/browser/_details/t5-data/experiments/scaling/sc-bi_v1-bsx4/operative_config.gin) # Parameters for AdafactorOptimizer: # ============================================================================== AdafactorOptimizer.beta1 = 0.0 AdafactorOptimizer.clipping\_threshold = 1.0 AdafactorOptimizer.decay\_rate = None AdafactorOptimizer.epsilon1 = 1e-30 AdafactorOptimizer.epsilon2 = 0.001 AdafactorOptimizer.factored = True AdafactorOptimizer.min\_dim\_size\_to\_factor = 128 AdafactorOptimizer.multiply\_by\_parameter\_scale = True ---
**Author:** ![moscow25](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moscow25/32/259_2.png) [@moscow25](https://discuss.huggingface.co/u/moscow25)\ **Post date:** [August 30, 2020, 3:37am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/15 "2020-08-30T03:37:13Z")
Thanks @saareliad for looking that up. I was going to do that, glad you did… So far for “scale by parameter size = True” – in my experiments 1. This does works. 2. The model converges much more slowly (for fine-tuning, on 8 GPU Volta) – and from what I saw to a worse number. But yes, it’s a parameter that does work. Maybe need to increase the LR or train for much longer with that on. ---
**Author:** ![moscow25](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/moscow25/32/259_2.png) [@moscow25](https://discuss.huggingface.co/u/moscow25)\ **Post date:** [August 30, 2020, 3:44am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/16 "2020-08-30T03:44:57Z")
As for FP16 – I did manage to get T5-Large checkpoints training in FP16… actually it won’t even inference (get NaN) for T5-Large without turning off feed-forward in FP16 or by doing some custom scaling magic… Even if you do only do FP16 on the attention operations and that’s it. I am able to fine-tune a checkpoint without NaN’s but the model diverges after a while. This happens even with just a small part of the model in FP16, and with a lower LR. It’s possible that this can eventually work, but it’s not simple. Moreover – if you are doing your attention operations in FP16 but saving all weights and gradients in FP32 (as well as FP16) – this may save a little bit of compute but does not save GPU memory at training. So this kind of very conservative FP16 is not useful, sad to say: 1. Diverges the model 2. Does not use less GPU memory (so can’t use larger batches or bigger model) 3. Does not train faster ---
**Author:** ![saareliad](https://avatars.discourse-cdn.com/v4/letter/s/3e96dc/32.png) [@saareliad](https://discuss.huggingface.co/u/saareliad)\ **Post date:** [August 30, 2020, 9:50am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/17 "2020-08-30T09:50:14Z")
What do you mean by “does work?” you clearly say that it made your results worse. From what I saw over T5-small “scale parameter” makes the result slightly worse as well. I’m currently trying T5-3B and the model doesn’t learn (beta1=0.9 and scale\_parameter=False) and default learning rate, so I wonder what can be done to mitigate it. ---
**Author:** ![DarkIntegral](https://avatars.discourse-cdn.com/v4/letter/d/839c29/32.png) [@DarkIntegral](https://discuss.huggingface.co/u/DarkIntegral)\ **Post date:** [September 2, 2020, 5:30pm UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/18 "2020-09-02T17:30:52Z")
TF32 Training on Ampere? There seems to be discussion (various threads and git issues) about whether T5 arch is just inherently unstable and that frequent FP16 NaN isn’t a bug in the transformers implementation or in user training arguments but may be unavoidable in true FP16 mode. I would like to bring into this discussion that when i run the mesh tensorflow version of T5 from the research repo ([https://github.com/google-research/text-to-text-transfer-transformer](https://github.com/google-research/text-to-text-transfer-transformer)) on TPU on my data set its rock solid 16 bit training (I assume because of the wider range capability of bf16 support). On the same data set I essentially can never get fp16 working on anything larger than t5-small with HuggingFace (with adafactor, with and without lr warming, native/apex(1/2/3) ect) For workflow reasons using the research mesh code is not going to be an option and I need to get the 3B model training on GPUs which will require ~16bit compute in order to fit in ~32-48GB gpu. Ampere gpus supports a similar size TF32 (18bit vs 16 bit for BF16 on TPU). Has anyone tried (or even have access to) an A100 GPU to see if TF32 solves the issue here? EDIT: it looks like Ampere also natively support BF16. So that looks like a good way to compare T5 mesh on TPU with T5 HF on Ampere both using BF16. ---
**Author:** ![jsrozner](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jsrozner/32/264_2.png) [@jsrozner](https://discuss.huggingface.co/u/jsrozner)\ **Post date:** [October 14, 2020, 9:19pm UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/20 "2020-10-14T21:19:22Z")
> task specific prefix doesn’t matter much. When finetuning on a task that is quite different from one of the supervised tasks, then: 1. Can you leave the default “summarization” task that finetune.py uses, or will this mess things up? 2. Should you create your own named task or just leave the task blank? (Does the model always expect some sort of \: prefix to every input?) ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 15, 2020, 8:51am UTC](https://discuss.huggingface.co/t/t5-finetuning-tips/684/21 "2020-10-15T08:51:56Z")
1. if the task is not related to “summarization” then it’ll probably mess thing up or slow down convergence, because the model will think it’s doing summarization because of the prefix. better to remove it. 2. Just iterating what I’ve said above. If your task is completely new and not related to one of the tasks on which T5 was trained then the prefix shouldn’t matter. [Next page](https://discuss.huggingface.co/t/t5-finetuning-tips/684.md?page=2)