# Training T5 on mlm task from scratch **URL:** \ **Category:** 🤗Transformers\ **Created:** [December 7, 2021, 4:00pm UTC](https://discuss.huggingface.co/t/training-t5-on-mlm-task-from-scratch/12617 "2021-12-07T16:00:26Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Arij](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@Arij](https://discuss.huggingface.co/u/Arij)\ **Post date:** [December 7, 2021, 4:00pm UTC](https://discuss.huggingface.co/t/training-t5-on-mlm-task-from-scratch/12617/1 "2021-12-07T16:00:26Z")
The main used reference is [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_t5_mlm_flax.py). I need to train T5 from hugging face from scratch on mlm task using pytorch. To my knowledge, there is no example to do that. The **_main issue_** that the same dataset preprocessing using the same T5 model but with two different frameworks flax and pytorch gave me different results. I did not change anything in the original run\_mlm\_flax.py code I just tried to use pytorch and Trainer instead. Everything is still as in the original code so why I am getting different results? I need torch version cause I have already built my model based on T5 from huggingface and I need also to train my model on mlm task and compare it with T5 from hugging face. That is why I started with T5 first as a baseline. I have decided as the first step to use [wikitext-103-raw-v1](https://huggingface.co/datasets/wikitext#wikitext-103-raw-v1) dataset for pretraining. The first question was in my mind which tokenizer to use so I have tried t5-small tokenizer to pretrain using the [original script,](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_t5_mlm_flax.py) then I trained the [tokenizer](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/t5_tokenizer_model.py) on train split of wikitext-103-raw-v1 dataset . 1. **First issue was using the pretrained tokenizer on wikitext-103-raw-v1 dataset gave me better results** and this raise another question in my mind , If I need to pretrain the model on mlm task then finetune it on another task, which tokenizer to use? I mean do I need to pretrain the tokenizer again and again evry time I will use new dataset? or simply uset 5-small tokenizer everywhere? or decide which datasets will be used in my experiements train the tokenizer on all train splits then do the pretraining and funetuning? 2. Second Issue : trying to mimic run\_mlm\_flax.py using torch Trainer keeping the dataset preprocessing and collator class with no change resulted in unsatisfied results even I tried to train on 100 epochs, still using 10 epochs with original script gives better results. Can you please guid me to the reason? I do not need the flax version I need torch pipeline to train T5 on mlm task from scratch. Seems my try was not good this is [my try](https://drive.google.com/file/d/1cjjlcy1-mr8gHji4wy0hMg2ZwX10fDqJ/view?usp=sharing) to run the same steps using pytorch. I have tried to use t5-small tokenizer. Also, I trained the given tokenizer in this repo on wikitext to compare. The results are not the same, seems strange. Training on 10 epochs using : 1. **if tokenizer trained on wiki** export CUDA\_VISIBLE\_DEVICES=0,1,2,3; python3 run\_t5\_mlm\_flax.py --output\_dir=“./ MLM-128wiki/wikitokenizer” --model\_type=“t5” --config\_name=”./wikitext-103-raw-v1" --tokenizer\_name=“./wikitext-103-raw-v1” --dataset\_name=“wikitext” --dataset\_config\_name=“wikitext-103-raw-v1” --max\_seq\_length=“128” --per\_device\_train\_batch\_size=“32” --per\_device\_eval\_batch\_size=“32” --adafactor --learning\_rate=“0.005” --weight\_decay=“0.001” --warmup\_steps=“2000” --overwrite\_output\_dir --logging\_steps=“500” --save\_steps=“10000” --eval\_steps=“500” --num\_train\_epochs=10 2. **if tokenizer is t5 -small tokenizer** export CUDA\_VISIBLE\_DEVICES=0,1,2,3; python3 run\_t5\_mlm\_flax.py --output\_dir=“./ MLM-128wiki/t5-tokenizer” --model\_type=“t5” --config\_name=”./wikitext-103-raw-v1" --tokenizer\_name=“t5-small” --dataset\_name=“wikitext” --dataset\_config\_name=“wikitext-103-raw-v1” --max\_seq\_length=“128” --per\_device\_train\_batch\_size=“32” --per\_device\_eval\_batch\_size=“32” --adafactor --learning\_rate=“0.005” --weight\_decay=“0.001” --warmup\_steps=“2000” --overwrite\_output\_dir --logging\_steps=“500” --save\_steps=“10000” --eval\_steps=“500” --num\_train\_epochs=10 **results** > ``` > T5tokenizer tokenizer trained on wiki > > ``` > > train loss: 2.307 ------ 2.074 > eval loss: 2.254 ------ 1.959 using my code as following: **1. if tokenizer trained on wiki:** export CUDA\_VISIBLE\_DEVICES=0,1,2,3; python3 rum\_mlm\_torch.py --output\_dir=“./torch/wiki” --model\_type=“t5” --config\_name=“./wikitext-103-raw-v1” --tokenizer\_name=“./wikitext-103-raw-v1” --dataset\_name=“wikitext” --dataset\_config\_name=“wikitext-103-raw-v1” --max\_seq\_length=“128” --per\_device\_train\_batch\_size=“32” --per\_device\_eval\_batch\_size=“32” --adafactor --learning\_rate=“0.005” --weight\_decay=“0.001” --warmup\_steps=“2000” --logging\_steps=“500” --save\_steps=“10000” --eval\_steps=“1000” --do\_train --do\_eval --do\_predict --overwrite\_output\_dir --report\_to=‘wandb’ --num\_train\_epochs=10 --evaluation\_strategy steps **2. if tokenizer is t5 tokenizer:** export CUDA\_VISIBLE\_DEVICES=0,1,2,3; python3 rum\_mlm\_torch.py --output\_dir=“./torch/t5tokenizer” --model\_type=“t5” --config\_name=“./wikitext-103-raw-v1” --tokenizer\_name=“t5-small” --dataset\_name=“wikitext” --dataset\_config\_name=“wikitext-103-raw-v1” --max\_seq\_length=“128” --per\_device\_train\_batch\_size=“32” --per\_device\_eval\_batch\_size=“32” --adafactor --learning\_rate=“0.005” --weight\_decay=“0.001” --warmup\_steps=“2000” --logging\_steps=“500” --save\_steps=“10000” --eval\_steps=“1000” --do\_train --do\_eval --do\_predict --overwrite\_output\_dir --report\_to=‘wandb’ --num\_train\_epochs=10 --evaluation\_strategy steps **results:** > ``` > T5tokenizer tokenizer trained on wiki > > ``` > > train loss: 4.675 ------ 3.961 > eval loss: 4.562 ------ 3.8 ---
**Author:** ![sadra](https://avatars.discourse-cdn.com/v4/letter/s/f17d59/32.png) [@sadra](https://discuss.huggingface.co/u/sadra)\ **Post date:** [February 8, 2022, 4:17pm UTC](https://discuss.huggingface.co/t/training-t5-on-mlm-task-from-scratch/12617/2 "2022-02-08T16:17:42Z")
Could you please share with us if you find anything related to this issue? ---
**Author:** ![Arij](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@Arij](https://discuss.huggingface.co/u/Arij)\ **Post date:** [July 25, 2022, 12:22pm UTC](https://discuss.huggingface.co/t/training-t5-on-mlm-task-from-scratch/12617/3 "2022-07-25T12:22:11Z")
Hi sadra, from my experiments I found that trained tokenizer always gives better results than using t5tokenizer for specific task. ---
**Author:** ![cmcmaster](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@cmcmaster](https://discuss.huggingface.co/u/cmcmaster)\ **Post date:** [July 29, 2022, 2:05am UTC](https://discuss.huggingface.co/t/training-t5-on-mlm-task-from-scratch/12617/4 "2022-07-29T02:05:27Z")
The main difference I see is that you’re using AdaFactor in the JAX script, but the default optimizer in the pytorch script (AdamW). ---
**Author:** ![Arij](https://avatars.discourse-cdn.com/v4/letter/a/45deac/32.png) [@Arij](https://discuss.huggingface.co/u/Arij)\ **Post date:** [July 29, 2022, 6:15am UTC](https://discuss.huggingface.co/t/training-t5-on-mlm-task-from-scratch/12617/5 "2022-07-29T06:15:29Z")
I am talking about comparision using the same optimizer, n the both experiements above I used adafactor)