# T5 for conditional generation: getting started **URL:** \ **Category:** Beginners\ **Created:** [September 28, 2020, 10:06pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284 "2020-09-28T22:06:07Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![jsrozner](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jsrozner/32/264_2.png) [@jsrozner](https://discuss.huggingface.co/u/jsrozner)\ **Post date:** [September 28, 2020, 10:06pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/1 "2020-09-28T22:06:07Z")
Hi, I have as specific task for which I’d like to use T5. Inputs look like some words \ some other words \ Training Outputs are a certain combination of the (some words) and (some other words). The goal is to have T5 learn the composition function that takes the inputs to the outputs, where the output should hopefully be good language. I was hoping to find an example script that I could modify. In particular I need a little help understanding how to do these parts: 1. When generating the input files (i.e. the mapping from input\_str to output\_str) what is the best format (e.g. a tsv for input and a tsv for output with a 1:1 mapping by line)? 2. Add special tokens to the vocab. Assuming that my inputs have special tokens and in the input files, then to make the model recognize them, I think I should use something like transformers.T5Tokenizer(additional\_special\_tokens=[,]). Is this correct? 3. Additional input processing: I think I need to somehow prepend a new “task tag” to all the input-output pairs. Where would I specify this new task name? : 4. Do I need to register this task somewhere so that it can actually be executed? Some of the examples I saw seem to suggest that I do. And do I need to choose a loss function for my new task? (If I don’t, will one be selected automatically?) 5. Any tips for the loss function? I care about the outputs being syntactical /grammatical, but I would also like the model to learn the relative positional relations of the inputs. For example, if I had something like a b c , the model might learn that abc, bac, cab, or cba are valid (i.e. in this case “a” and “b” must always be adjacent), and would choose the sequence that is most probable under the language model. Thanks! ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [September 29, 2020, 9:30am UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/2 "2020-09-29T09:30:03Z")
1. You can choose whatever format that works well for you, only thing to note is your `dataset` or `collator`should return `input_ids`, `attention_mask` and `labels`. 2. To add new tokens ```pyhton tokenizer.add_tokens(list_of_new_tokens) # resize the embeddings model.resize_token_embeddings(len(tokenizer)) ``` 1. Using task prefix is optional. 2. No, you won’t need to register the task, the original `T5` repo requires that but it’s not required here. You might find these two notebooks useful 1. [Fine-tune T5 for Classification and Multiple Choice](https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb) 2. [Fine-tune T5 for Summarization](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb) 3. [Train T5 on TPU](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb) > Note: These notebooks manually add the `eos` token (``), but it’s not with the current version, the tokenizer will handle that. Here’s a great thread on tips and tricks for T5 fine-tuning > [@T5 Finetuning Tips](https://discuss.huggingface.co/t/t5-finetuning-tips/684): > > Starting this for results, sharing + tips and tricks, and results. This is my first attempt at this kind of thread so it may completely fail. Some things I’ve found Apparently if you copy AdaFactor from fairseq, as recommended by t5 authors, you can fit batch size = 2 for t5-large lm finetuning fp16 rarely works. for most tasks, you need to manually add \ to the end of your sequence. Thing’s I’ve read task specific prefix doesn’t matter much. cc @mrm8488 @valhalla @patrickvonplaten who… ---
**Author:** ![jsrozner](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jsrozner/32/264_2.png) [@jsrozner](https://discuss.huggingface.co/u/jsrozner)\ **Post date:** [September 29, 2020, 10:16pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/3 "2020-09-29T22:16:15Z")
Thank you! The linked notebooks seem to do a lot of boilerplate work that is now handled in the seq2seq finetune.py script? After spending a few hours reading through code, it seems like I should be able to finetune by just 1. updating the tokens as you’ve described 2. creating a dataset (train.source, train.target, etc) 3. running finetune.sh with those as parameters Is this correct? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [September 30, 2020, 3:17am UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/4 "2020-09-30T03:17:42Z")
Yes, most of the code is similar to `finetune.py` To use `finetune.py` for your use case without modifying it - 1. add the tokens, 2. resize the embeddings of the model 3. save this model and pass it `finetune.py` ---
**Author:** ![jsrozner](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jsrozner/32/264_2.png) [@jsrozner](https://discuss.huggingface.co/u/jsrozner)\ **Post date:** [September 30, 2020, 3:46am UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/5 "2020-09-30T03:46:26Z")
Cool, yes this is what I was able to accomplish! What would be a good reason for not re-using finetune.py? Or in the case that someone wants more customization, it would make sense to just tweak the finetune.py file? Yesterday it took me a while to find the directory examples/seq2seq. I’m curious: why aren’t the modules (e.g. SummarizationModule) available in a higher level transformers directory? (e.g. something like transformers/seq2seq or transformers/LMgeneration)? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [September 30, 2020, 2:42pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/6 "2020-09-30T14:42:21Z")
You can use `finetune.py` as it is or modify as required, no strong reason to not use it. The shared notebooks were written before `finetune.py`. all examples can be found under `examples/` ---
**Author:** ![jsrozner](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jsrozner/32/264_2.png) [@jsrozner](https://discuss.huggingface.co/u/jsrozner)\ **Post date:** [November 11, 2020, 3:54am UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/7 "2020-11-11T03:54:37Z")
In case anyone else comes across this thread, I adopted a self-contained T5 finetune example that doesn’t use lightning. [T5 Finetune on github](https://github.com/jsrozner/t5_finetune/blob/main/train.py) ---
**Author:** ![Savindu](https://avatars.discourse-cdn.com/v4/letter/s/e9a140/32.png) [@Savindu](https://discuss.huggingface.co/u/Savindu)\ **Post date:** [August 1, 2021, 5:09pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/8 "2021-08-01T17:09:15Z")
Hi, I used your GitHub code for finetune the T5 for text generation. **I have a issue of partially generating the output.** I don’t know why the output is cropped. For example this is the generated text: “\< pad \> Kasun has 7 books and gave Nimal 2 of the books. How many book did Ka” This is the full output. - This is not the full sentence and not end with \< /s \> token. - Output are maximum 17 words. Giving model.generate( . . . , min\_length=30 ) is also not solved this issue. - only small amount of word contains sentence are giving full output sentence Parameters I used: max\_src\_len = 200 max\_tgt\_len = 512 Is that cause because of a parameter value which I pass to tokenizer or decoder? ---
**Author:** ![jsrozner](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jsrozner/32/264_2.png) [@jsrozner](https://discuss.huggingface.co/u/jsrozner)\ **Post date:** [August 1, 2021, 6:25pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/9 "2021-08-01T18:25:47Z")
You should add `max_length=None` to your model.generate() call, I think. If that doesn’t work, try `max_length=500` or something and see if generations are longer. I think you should also set `min_length=None`. The reason is that T5forConditionaGeneration I think loads a config file at some point that specifies these parameters. You can see default value at [transformers/generation\_utils.py at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L697) So if you want to see what the model is being loaded with when we do `.frompretrained()`, call `print(model.config)`. I think we’ll see that the default is max\_length=20, which would be causing your problem. Set both max length and min length to None, and then the model will stop only when EOS token is the most probable output. edit: I think you could also directly modify some of these config parameters at load, e.g. by `model.config.max_length = new_value`, rather than doing it at the generation call. ---
**Author:** ![Savindu](https://avatars.discourse-cdn.com/v4/letter/s/e9a140/32.png) [@Savindu](https://discuss.huggingface.co/u/Savindu)\ **Post date:** [August 1, 2021, 7:02pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/10 "2021-08-01T19:02:22Z")
@jsrozner Thank you so much. ---
**Author:** ![pafitis](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pafitis/32/2269_2.png) [@pafitis](https://discuss.huggingface.co/u/pafitis)\ **Post date:** [August 4, 2021, 12:36pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/11 "2021-08-04T12:36:27Z")
Hi @Savidhu, does it actually work for you? Setting min\_length / max\_length doesn’t seem to change the output for me. ---
**Author:** ![Savindu](https://avatars.discourse-cdn.com/v4/letter/s/e9a140/32.png) [@Savindu](https://discuss.huggingface.co/u/Savindu)\ **Post date:** [August 4, 2021, 6:11pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/12 "2021-08-04T18:11:05Z")
@pafitis Yes that actually worked. I changed the configuration as @jsrozner mentioned. My settings are: `* Max_length = None` `* Min_length = None` for the model.generate() `* model.config.max_length = 50` You need to change `model.config.max_length` also. ---
**Author:** ![prikmm](https://avatars.discourse-cdn.com/v4/letter/p/a698b9/32.png) [@prikmm](https://discuss.huggingface.co/u/prikmm)\ **Post date:** [August 8, 2021, 7:33pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/13 "2021-08-08T19:33:14Z")
Hey all, I have been trying to finetune T5 on XSum and I am getting constant validation loss. It doesn’t change at all. The training loss varies a but doesn’t converge like it stays in the range [10.0, 12.0]. I tried many methods like creating my own nn.Module which compatible with `Trainer()`, etc but none worked. [Link](https://colab.research.google.com/drive/1KEweUQA8LRk_5VyRfAt04a7g1g6GpgrP?usp=sharing) to colab (first version where I used default `Trainer()`). Can anyone share a colab link or wandb project for my reference? Thanks! ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [August 20, 2021, 2:30pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/14 "2021-08-20T14:30:28Z")
Hi, I have a Colab notebook on how to fine-tune T5 on a custom dataset here: [GitHub - NielsRogge/Transformers-Tutorials: This repository contains demos I made with the Transformer](https://github.com/NielsRogge/Transformers-Tutorials) Let me know if it helps. ---
**Author:** ![phade160](https://avatars.discourse-cdn.com/v4/letter/p/bb73d2/32.png) [@phade160](https://discuss.huggingface.co/u/phade160)\ **Post date:** [March 25, 2022, 4:48pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/15 "2022-03-25T16:48:47Z")
Hi, I am finetuning a T5 model for QA on my dataset but the vocab is so different than the tokenizer’s, which results in an excessive length of token\_ids/tokens. can I train a new tokenizer from the existing one and use it for finetuning? if yes, any tips/resources to aid? Thanks ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [June 9, 2023, 4:35am UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/16 "2023-06-09T04:35:01Z")
hi @valhalla , @jsrozner my subject is that, I have some challenges and for each challenge I have a related solution. I want to do solution generation. means the input would be challenge and the output would be solution. I applied gpt-neo which the results is not very well. do you think t5 conditional generation can be a good try? I am not familiar what condition means here? thanks for your guidence. Iam thinking about adding some condition but don’t know if T5 is a good option? what is the most application of T5? ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [June 9, 2023, 4:35am UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/17 "2023-06-09T04:35:56Z")
Hi @nielsr my subject is that, I have some challenges and for each challenge I have a related solution. I want to do solution generation. means the input would be challenge and the output would be solution. I applied gpt-neo which the results is not very well. do you think t5 conditional generation can be a good try? I am not familiar what condition means here? thanks for your guidence. Iam thinking about adding some condition but don’t know if T5 is a good option? ---
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [July 17, 2023, 2:26pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/18 "2023-07-17T14:26:56Z")
Hi, Yes T5 could be an option. I’d advise to use [Flan-T5](https://huggingface.co/docs/transformers/model_doc/flan-t5), which is a better trained version of T5. ---
**Author:** ![SUNM](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SUNM](https://discuss.huggingface.co/u/SUNM)\ **Post date:** [July 17, 2023, 3:54pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/19 "2023-07-17T15:54:46Z")
Thanks for reply ---
**Author:** ![asz123](https://avatars.discourse-cdn.com/v4/letter/a/c37758/32.png) [@asz123](https://discuss.huggingface.co/u/asz123)\ **Post date:** [July 17, 2023, 7:36pm UTC](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284/20 "2023-07-17T19:36:50Z")
I am also a beginner starting to use T5 for question answering but not extractive from my own dataset. I have it loaded into a data set with two keys “inputs” - which make up context and questions and “targets” which are the answers. Do I need a labels and index section and also I am a little unclear on where to go from here any advice or tips would be greatly appreciated. [Next page](https://discuss.huggingface.co/t/t5-for-conditional-generation-getting-started/1284.md?page=2)