# Does task specific prefix matters for T5 fine-tuning? **URL:** \ **Category:** Beginners\ **Created:** [July 28, 2020, 4:57pm UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501 "2020-07-28T16:57:46Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 28, 2020, 4:57pm UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/1 "2020-07-28T16:57:47Z")
If I understand correctly pre-trained T5 models were pre-trained with an unsupervised objective without any task specific prefix like “translate”, “summarize”, etc. Is it important then to create my summarization dataset for fine-tuning in a way that every input starts with "summarize: "? ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [July 28, 2020, 7:27pm UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/2 "2020-07-28T19:27:59Z")
I think it is important, but am not totally certain why. You could test it pretty easily I bet. ---
**Author:** ![mengyahu](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@mengyahu](https://discuss.huggingface.co/u/mengyahu)\ **Post date:** [July 29, 2020, 2:20am UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/3 "2020-07-29T02:20:44Z")
Actually, it may not be very important. I am confused about the prefix too and ask on the Github. And one of the code author replied. (there are more questions, you may benefit from it too as a beginner) [https://github.com/huggingface/transformers/issues/6007](https://github.com/huggingface/transformers/issues/6007) ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 29, 2020, 5:44am UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/4 "2020-07-29T05:44:14Z")
I think the question is if the released pre-trained models were trained on a mixture of supervised and unsupervised tasks (so they learned with prefixes and putting them in front of my input when fine-tuning is useful) or only on unsupervised tasks. I know they do it both ways in the paper but not clear which one was released. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [July 29, 2020, 6:07am UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/5 "2020-07-29T06:07:12Z")
Use prefix if your task is similar to one of the pre-trained tasks, otherwise it really doesn’t matter (at least I have observed this in my own experiments) ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 29, 2020, 4:30pm UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/6 "2020-07-29T16:30:51Z")
Released models were pret-rained on both supervised and unsupervised tasks so prefix should help. See [https://github.com/google-research/text-to-text-transfer-transformer/issues/314#issuecomment-665764712](https://github.com/google-research/text-to-text-transfer-transformer/issues/314#issuecomment-665764712) ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [July 29, 2020, 4:32pm UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/7 "2020-07-29T16:32:26Z")
Yes, that’s what I said, if task is similar to one of the tasks used in the pre-training mixture then it helps, if it’s completely different then it won’t matter ---
**Author:** ![marton-avrios](https://avatars.discourse-cdn.com/v4/letter/m/ecae2f/32.png) [@marton-avrios](https://discuss.huggingface.co/u/marton-avrios)\ **Post date:** [July 29, 2020, 4:35pm UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/8 "2020-07-29T16:35:37Z")
Yes, that part was clear. I just wanted to clarify how the released models were trained. Although you are right, the answer was implicit in what you already said. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [July 29, 2020, 4:40pm UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/9 "2020-07-29T16:40:59Z")
Great part is that T5 performs really well with and without prefix 😄 Here’s what I observed in my experiments., 1. It converges slightly faster when using a task prefix and when the task was similar, say summarization 2. Performed equally well even without prefix, took slightly longer to converge ---
**Author:** ![Arij](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@Arij](https://discuss.huggingface.co/u/Arij)\ **Post date:** [June 28, 2021, 7:10am UTC](https://discuss.huggingface.co/t/does-task-specific-prefix-matters-for-t5-fine-tuning/501/10 "2021-06-28T07:10:37Z")
My question is related, how can I understand how prefix and prefix masking is working? where can I read the code related to processing the prefix? or there is nothing special about that? I mean it is simply adding a task in the config file? is the idea as simple as that?