# Pretraining T5 from scratch using MLM

**URL:** <https://discuss.huggingface.co/t/pretraining-t5-from-scratch-using-mlm/73360>\
**Category:** Models\
**Created:** [February 15, 2024, 5:23pm UTC](https://discuss.huggingface.co/t/pretraining-t5-from-scratch-using-mlm/73360 "2024-02-15T17:23:18Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![OmarHaroon01](https://avatars.discourse-cdn.com/v4/letter/o/db5fbb/32.png) [@OmarHaroon01](https://discuss.huggingface.co/u/OmarHaroon01)\
**Post date:** [February 15, 2024, 5:23pm UTC](https://discuss.huggingface.co/t/pretraining-t5-from-scratch-using-mlm/73360/1 "2024-02-15T17:23:18Z")

</div>

Hello everyone, I am new to pretraining language models. I have a query regarding pretraining T5 using MLM. I used around 700 thousands tokens (around 20000 rows from the wikipedia.simple dataset) to pretrain t5-small for MLM but my loss never goes down below 4.5. I trained for 15 epochs. The inference is also not good when I try to predict a masked token. Can someone please help me with what the issue is here? It would mean alot.

My code is provided below  
[Pretrain\_t5](https://colab.research.google.com/drive/1vxFf9VkPGaEZBjjfaBUdLKxFouQ6cw1I?usp=sharing)

Most of the code has been picked from transformers example of pretraining

---

<div class="post-metadata">

**Author:** ![sxdmit](https://avatars.discourse-cdn.com/v4/letter/s/82dd89/32.png) [@sxdmit](https://discuss.huggingface.co/u/sxdmit)\
**Post date:** [December 6, 2024, 11:58am UTC](https://discuss.huggingface.co/t/pretraining-t5-from-scratch-using-mlm/73360/2 "2024-12-06T11:58:55Z")

</div>

hello, i am pertrain t5 many times, after carefuly checking your setting , i believe may be is due to a higher lr, please make it smaller.