# Incremental training on unlabeled data using MLM

**URL:** <https://discuss.huggingface.co/t/incremental-training-on-unlabeled-data-using-mlm/27604>\
**Category:** 🤗Transformers\
**Created:** [December 10, 2022, 2:07pm UTC](https://discuss.huggingface.co/t/incremental-training-on-unlabeled-data-using-mlm/27604 "2022-12-10T14:07:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![bitsanlp](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bitsanlp/32/12988_2.png) [@bitsanlp](https://discuss.huggingface.co/u/bitsanlp)\
**Post date:** [December 10, 2022, 2:07pm UTC](https://discuss.huggingface.co/t/incremental-training-on-unlabeled-data-using-mlm/27604/1 "2022-12-10T14:07:26Z")

</div>

I have an unlabeled data dump of 2 million sentences.

I have fine-tuned a roberta-base model with masked language modeling on the first 100k sentences using the implementation described by Lewis Tunstall in [this notebook](https://github.com/nlp-with-transformers/notebooks/blob/main/09_few-to-no-labels.ipynb).

Now I want to try to fine-tune the roberta-base model on 250k sentences to compare the effect of training on larger data on a downstream, binary classification task but I have limited compute (Google colab).

I want to know whether the below two approaches result in the same model or not:

1. fine-tuned a roberta-base model with masked language modeling on the first 250k sentences from scratch

2. fine-tuned the model already trained on 100k sentences on the next 150k sentences with masked language modeling

In theory, I believe both should give me the exact same final model. But I want to verify.  
Second approach would save me half the compute needed for the first.