# Tips for PreTraining BERT from scratch

**URL:** <https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175>\
**Category:** 🤗Transformers\
**Created:** [September 18, 2020, 3:37am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175 "2020-09-18T03:37:22Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [September 18, 2020, 3:37am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/1 "2020-09-18T03:37:22Z")

</div>

So far, I’ve been using pre-trained models. For my task, it seems that I am required to perform pre-training on GLUE task just to see how it performs. I wanted to confirm what modifications need to be done to do this ? I’m not sure about using the same tokenizer.

I want to randomly initialize it and train on GLUE task. Additionally, if you some tips on effectively doing it when not using fine-tuning weights, please share ?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 18, 2020, 4:33am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/2 "2020-09-18T04:33:40Z")

</div>

You can initialize a model without pre-trained weights using

```auto
from transformers import BertConfig, BertForSequenceClassification

# either load pre-trained config
config = BertConfig.from_pretrained("bert-base-cased")
# or instantiate yourself
config = BertConfig(
    vocab_size=2048,
    max_position_embeddings=768,
    intermediate_size=2048,
    hidden_size=512,
    num_attention_heads=8,
    num_hidden_layers=6,
    type_vocab_size=5,
    hidden_dropout_prob=0.1,
    attention_probs_dropout_prob=0.1,
    num_labels=3,
)

# pass the config to model constructor instead of from_pretrained
# this creates the model as per the params in config 
# but with weights randomly initialized
model = BertForSequenceClassification(config)

```

and as it’s a `ForSequenceClassification` model, the existing `run_glue.py` script can be used to train this model, just initialize model using `config` instead of `.from_pretrained`

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [September 18, 2020, 4:34am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/3 "2020-09-18T04:34:34Z")

</div>

Thanks for replying. Okay, so the process is mostly same. Do we need to make any changes to tokenizer ? Saw few posts where people had encountered issues during pre-training, so thought of confirming this.

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [September 18, 2020, 4:35am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/4 "2020-09-18T04:35:46Z")

</div>

Also any training tips when not using pre-trained weights ?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 18, 2020, 4:36am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/5 "2020-09-18T04:36:22Z")

</div>

If you train the tokenizer from scratch as well, then make sure to change the vocab size in `config` accordingly

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 18, 2020, 4:41am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/6 "2020-09-18T04:41:27Z")

</div>

I haven’t done so myself for this task so can’t say much, but probably start with higher LR than the default one in the `Trainer` (which is 5e-5) as we are training from scratch, experiment with LR schedule, haparms search will definitely help choose the right params.

@sgugger might have better tips for this.

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [September 18, 2020, 4:46am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/7 "2020-09-18T04:46:32Z")

</div>

Is there any problem if we use `AutoTokenizer.from_pretrained()` ? That’s where I am unsure. Or do we use custom tokenizer ? What’s the recommended way of proceeding.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 18, 2020, 4:50am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/8 "2020-09-18T04:50:06Z")

</div>

You can use pre-trained tokenizer, it shouldn’t cause any issues. And IMO using pre trained tokenizer makes sense than training from scratch on limited data

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [September 18, 2020, 8:49am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/9 "2020-09-18T08:49:58Z")

</div>

With the standard way, BERT-base is training, BERT-large doesn’t seem to respond. I seem to be missing something in regards to training dynamics.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 18, 2020, 9:21am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/10 "2020-09-18T09:21:30Z")

</div>

is there any exception , error ?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [September 18, 2020, 9:56am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/11 "2020-09-18T09:56:00Z")

</div>

Just to add to this, training your own tokenizer is mainly useful if you are working with a specific genre, domain, and/or language.

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [September 18, 2020, 10:16am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/12 "2020-09-18T10:16:26Z")

</div>

No, it doesn’t learn anything. Tried with different LRs. Are there certain specific things involved when training from scratch with these large models ? BERT-base improves from 31 to 58, whereas BERT-large stays at 31.

---

<div class="post-metadata">

**Author:** ![vgoklani](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vgoklani/32/16670_2.png) [@vgoklani](https://discuss.huggingface.co/u/vgoklani)\
**Post date:** [September 18, 2020, 6:50pm UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/13 "2020-09-18T18:50:14Z")

</div>

More generally, what sort of performance metrics should one expect to see when pre-training?

To be more specific, how long should we pre-train (days vs weeks), and what’s an acceptable loss? When should we stop? I would appreciate any references on this issue, thank you!

---

<div class="post-metadata">

**Author:** ![RichardWang](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/richardwang/32/45198_2.png) [@RichardWang](https://discuss.huggingface.co/u/RichardWang)\
**Post date:** [September 20, 2020, 2:18am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/14 "2020-09-20T02:18:54Z")

</div>

Perform pretraining on GLUE tasks with non-pretrained large model ?

I don’t know but multi-task learning may help ?  
Additionally, since dataset size is smaller than those large text corpus for pretraining, stronger regularization (dropout, weight decay, gradient clip…) may help.

---

<div class="post-metadata">

**Author:** ![RichardWang](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/richardwang/32/45198_2.png) [@RichardWang](https://discuss.huggingface.co/u/RichardWang)\
**Post date:** [September 20, 2020, 2:27am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/15 "2020-09-20T02:27:51Z")

</div>

> [@vgoklani](#):
>
> what sort of performance metrics should one expect to see when pre-training?

Typically GLUE / SuperGLUE in English, or finetuning on your target domain should be also ok for a model targeted for that domain I think.

> [@vgoklani](#):
>
> how long should we pre-train

[Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361) should help. In short, it depends on the size of model and how much cost you can afford or how much performance you want. I remembered huggingface has a calculator for this.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 24, 2020, 2:16pm UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/16 "2020-09-24T14:16:23Z")

</div>

Hey @prajjwal1 were you able to resolve this ?

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [September 24, 2020, 2:31pm UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/17 "2020-09-24T14:31:29Z")

</div>

I’ve opened a new [issue](https://discuss.huggingface.co/t/pre-train-bert-without-nsp/1245) which is about pre-training. Training on GLUE part is resolved. Thanks for asking.

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 24, 2020, 3:19pm UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/18 "2020-09-24T15:19:12Z")

</div>

Glad that it’s resolved. What kind of metrics are you getting ?

---

<div class="post-metadata">

**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\
**Post date:** [September 25, 2020, 5:27am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/19 "2020-09-25T05:27:02Z")

</div>

After experimenting, I felt as if I had replicated results of Table 1 from Revealing Dark Secrets of BERT paper.

---

<div class="post-metadata">

**Author:** ![danaludwig](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/danaludwig/32/393_2.png) [@danaludwig](https://discuss.huggingface.co/u/danaludwig)\
**Post date:** [December 10, 2020, 4:41am UTC](https://discuss.huggingface.co/t/tips-for-pretraining-bert-from-scratch/1175/20 "2020-12-10T04:41:29Z")

</div>

Take this with a grain of salt, but I heard that BERT-large can’t be trained without a TPU because it has too many parameters to fit into GPU memory.