# Doing classification 100% from scratch?

**URL:** <https://discuss.huggingface.co/t/doing-classification-100-from-scratch/9699>\
**Category:** 🤗Transformers\
**Created:** [September 3, 2021, 12:57am UTC](https://discuss.huggingface.co/t/doing-classification-100-from-scratch/9699 "2021-09-03T00:57:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\
**Post date:** [September 3, 2021, 12:57am UTC](https://discuss.huggingface.co/t/doing-classification-100-from-scratch/9699/1 "2021-09-03T00:57:22Z")

</div>

Hello there!

I have a quite specific corpus and I wanted to try an approach where I do everything from scratch (well, almost!). Specifically, I was thinking about:

1. training a `language model` from scratch using my own corpus and creating my own `tokenizer`. Hugginface provides two colab for that (see [Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/language_modeling_from_scratch.ipynb#scrollTo=n2ZRs1cL3l-R) for the LM)

2. Then, simply loading the language model created at 1. and fine-tuning it for text classification using the usual imports

```auto
from transformers import AutoModelForSequenceClassification 
model = AutoModelForSequenceClassification.from_pretrained("my_language_model")
#do some classification
#save final model

```

Does that make sense? Are these the right conceptual steps?  
Thanks!

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 3, 2021, 12:22pm UTC](https://discuss.huggingface.co/t/doing-classification-100-from-scratch/9699/2 "2021-09-03T12:22:45Z")

</div>

Yes, that are the right steps: first pre-training, than fine-tuning a head. However, note that you need a relatively big corpus in order for pre-training to be effective. If you really have some in-domain data that’s different from the corpora that were used to pre-train models like BERT and RoBERTa, then it might be useful to do it.

Examples are BioBERT (pre-trained on biomedical language), SciBERT (pre-trained on scientific text), etc.

---

<div class="post-metadata">

**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\
**Post date:** [September 3, 2021, 12:56pm UTC](https://discuss.huggingface.co/t/doing-classification-100-from-scratch/9699/3 "2021-09-03T12:56:33Z")

</div>

thanks @nielsr, I saw this myriad of yourmodelBERT models. I just wonder: is BERT necessarily the best architecture for training a language model from scratch? Why have people focused on this particular model (which is almost an old model by today’s standards)

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 3, 2021, 2:50pm UTC](https://discuss.huggingface.co/t/doing-classification-100-from-scratch/9699/4 "2021-09-03T14:50:06Z")

</div>

BERT is the one that started it all, and it still works really well. All other model variants (cf. DeBERTa, RoBERTa, ConvBERT, Funnel Transformer, etc.) are just small tweaks to the original BERT architecture. They are fancy but [research has shown](https://arxiv.org/abs/2102.11972) that none of them actually improves upon the original Transformer by a large margin.

---

<div class="post-metadata">

**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\
**Post date:** [September 17, 2021, 4:25pm UTC](https://discuss.huggingface.co/t/doing-classification-100-from-scratch/9699/5 "2021-09-17T16:25:35Z")

</div>

@nielsr just a follow up if you have a moment. The TF notebook for language modeling actually mention two different tasks: causal language modeling and masked language modeling.

For the purpose of training a classifier on top of the model I train from scratch, are the two basic tasks equivalent? That is I can train a causal language modeling and then train a classifier with it or I can train a masked language model and then train the classifier. Are both approaches OK conceptually?

Thanks!