# Fine tuning T5 Encoder and T5 Decoder separately

**URL:** <https://discuss.huggingface.co/t/fine-tuning-t5-encoder-and-t5-decoder-separately/14437>\
**Category:** 🤗Transformers\
**Created:** [February 5, 2022, 4:46pm UTC](https://discuss.huggingface.co/t/fine-tuning-t5-encoder-and-t5-decoder-separately/14437 "2022-02-05T16:46:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ND1](https://avatars.discourse-cdn.com/v4/letter/n/c0e974/32.png) [@ND1](https://discuss.huggingface.co/u/ND1)\
**Post date:** [February 5, 2022, 4:46pm UTC](https://discuss.huggingface.co/t/fine-tuning-t5-encoder-and-t5-decoder-separately/14437/1 "2022-02-05T16:46:58Z")

</div>

Is it possible to create instances for T5Encoder and T5Decoders separately. I would like to fine tune T5Encoder with Masked Lang Objective on company specific dataset and then use it with T5Decoder for text generation.  
I have tried and searched exclusively on Huggingface for these classes and/or their documentation but could not find so.

---

<div class="post-metadata">

**Author:** ![Saptarshi7](https://avatars.discourse-cdn.com/v4/letter/s/9e8a1a/32.png) [@Saptarshi7](https://discuss.huggingface.co/u/Saptarshi7)\
**Post date:** [May 6, 2024, 10:05pm UTC](https://discuss.huggingface.co/t/fine-tuning-t5-encoder-and-t5-decoder-separately/14437/2 "2024-05-06T22:05:27Z")

</div>

Hey (@ND1 ), did you find anything on this?.. I tried looking at the same but couldn’t find anything. I mean, we can extract the T5 encoder and decoder separately. However, trying to tune them with AutoModelForMaskedLM is not possible I guess.

EDIT…

I think I was right. T5 is a Seq-2-Seq model, which means that for language modelling at least, it needs to be trained in that fashion.

Another reason why you can’t train the encoder/decoder separately for LM is that, for the encoder, you’d likely want to use AutoModelForMaskedLM but T5 is not supported by this class. This makes sense if you think about it.

MLM models like BERT use [MASK] tokens whereas T5’s encoder uses _span replacement tokens_ which are slightly different than [MASK].

So, I would recommend using this script: [transformers/examples/flax/language-modeling/run\_t5\_mlm\_flax.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/examples/flax/language-modeling/run_t5_mlm_flax.py)

which will get you up and running with training T5 in S2S mode. Then, you can throw away the decoder and use the encoder for downstream tasks.

Hope this helps 🙂