# Pre-Train BERT (from scratch) **URL:** \ **Category:** Research\ **Created:** [September 24, 2020, 1:01pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245 "2020-09-24T13:01:31Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 24, 2020, 1:01pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/1 "2020-09-24T13:01:31Z")
BERT has been trained on MLM and NSP objective. I wanted to train BERT with/without NSP objective (with NSP in case suggested approach is different). I haven’t performed pre-training in full sense before. Can you please share how to obtain the data (crawl and tokenization details which were used) on which BERT was trained on ?. Since it takes a lot of time, I am looking for well tested code that can yield the BERT with/without NSP in one go. Any suggestions will be helpful. I know about some projects like [these](https://github.com/NVIDIA/Megatron-LM/tree/main/examples), but they won’t integrate well with `transformers` well I guess which is a must have condition in my case. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [September 25, 2020, 6:44am UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/3 "2020-09-25T06:44:43Z")
BERT was trained on `book corpus` and `english wikipedia` both of which are available in `dataset` library > **[wikipedia · Datasets at Hugging Face](https://huggingface.co/datasets/wikipedia)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[bookcorpus · Datasets at Hugging Face](https://huggingface.co/datasets/bookcorpus)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. Transformers has recently included dataset for for next sent prediction which you could use > and there’s also NSP head for BERT [https://github.com/huggingface/transformers/blob/master/src/transformers/modeling\_bert.py#L560](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L560) **EDIT** : `BertForPreTraining` class can be used for both `MLM` and `NSP` with the current `example/languae-modeling` I guess it’s only possible to either use `MLM` or `NSP`, you might need to write your own script to combine these. ---
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 25, 2020, 7:39am UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/4 "2020-09-25T07:39:48Z")
For training on MLM objective, is it recommended to use `collate_fn` from [here](https://github.com/huggingface/transformers/blob/6303b5a7185fba43830db0cbb06c61861f57ddff/src/transformers/data/data_collator.py#L120)? Didn’t see `TextDataset` for MLM objective. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [September 25, 2020, 7:42am UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/5 "2020-09-25T07:42:53Z")
Masking is done using `DataCollatorForLanguageModeling` so you can use any dataset and just pass the collator to `DataLoader`. **One thing to note:** `DataCollatorForLanguageModeling` does dynamic masking but BERT was trained using static masking . ---
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 25, 2020, 7:52am UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/6 "2020-09-25T07:52:20Z")
It seems that using `BertForNextSentencePrediction` with `TextDatasetForNextSentencePrediction` and `DataCollatorForLanguageModeling` would be equivalent to the BERT objective (except static masking part). And for dataset, we can use `datasets.concatenate_datasets()` method for BookCorpus and Wikipedia. This might be close right ? Any additional details ? ---
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 25, 2020, 9:10am UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/7 "2020-09-25T09:10:05Z")
`datasets.concatenate_datasets()` does not seem to work for this since features do not match. Also `BertForNextSentencePrediction` expects a `file_path`. Initially I thought it was a wrapper which can take `datasets` objects. ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [September 25, 2020, 10:25am UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/8 "2020-09-25T10:25:43Z")
It shouldn’t be hard to convert `BertForNextSentencePrediction` to use datasets. I played with wikipedia dataset for english just now. Each dataset entry is an article/document and it needs to be sentence tokenized in `BertForNextSentencePrediction` . Book corpus dataset entries seem to be sentences already. Let me know about your progress. ---
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 25, 2020, 10:27am UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/9 "2020-09-25T10:27:59Z")
How are you measuring the metric ? ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [September 25, 2020, 10:39am UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/10 "2020-09-25T10:39:47Z")
I don’t yet. I am still setting up these training pipelines. I asked about metrics at [Evaluation metrics for BERT-like LMs](https://discuss.huggingface.co/t/evaluation-metrics-for-bert-like-lms/1256) but no response yet. I read at [https://huggingface.co/transformers/perplexity.html](https://huggingface.co/transformers/perplexity.html) and elsewhere that perplexity is not appropriate for BERT and MLMs. Can’t we use fill-mask pipeline and some version of masking accuracy? OTOH, I’ve already setup GLUE benchmarks with [https://jiant.info/](https://jiant.info/) v2 Alpha. Excellent integration with transformers and can easily plugin any model and run benchmarks in parallel. See [https://github.com/jiant-dev/jiant/tree/master/examples](https://github.com/jiant-dev/jiant/tree/master/examples) for more details ---
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 25, 2020, 10:44am UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/11 "2020-09-25T10:44:05Z")
Did you try using Cross Entropy for pre-training ? We usually use that for MLM. It can be easily used for NSP I guess. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [September 25, 2020, 1:29pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/12 "2020-09-25T13:29:11Z")
Indeed wikipedia has columns “text” and “title” while bookcorpus only has “text”. You can concatenate them by removing the “title” column from wikipedia: ```python from datasets import load_dataset, concatenate_datasets wiki = load_dataset("wikipedia", "20200501.en", split="train") bookcorpus = load_dataset("bookcorpus", split="train") print(wiki.column_names, bookcorpus.column_names) # ['title', 'text'] ['text'] wiki.remove_columns_("title") bert_dataset = concatenate_datasets([wiki, bookcorpus]) ``` ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [September 25, 2020, 1:33pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/13 "2020-09-25T13:33:32Z")
Let me know if you find an appropriate way to cut wikipedia articles into sentences ! Also don’t hesitate if you have any questions about dataset processing, I’d be happy to help 🙂 ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [September 25, 2020, 2:34pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/14 "2020-09-25T14:34:09Z")
You can use spaCy or stanza for sentence segmentation. spaCy is quite a bit faster but might be less correct. If you want to I can post a segmentation function here. ---
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 25, 2020, 2:36pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/15 "2020-09-25T14:36:58Z")
So after concatenation of wikipedia and book\_corpus, next things to do is NSP. Can you suggest how that is to be done on object after concatenation happens? I do not want to diverge from the actual method which was used to pre-train BERT. ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [September 25, 2020, 2:39pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/16 "2020-09-25T14:39:24Z")
You can have a look here: > ---
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 25, 2020, 2:39pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/17 "2020-09-25T14:39:55Z")
Has anyone replicated BERT pre-training from scratch ? It would be good to hear what exactly did they do. ---
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 25, 2020, 2:40pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/18 "2020-09-25T14:40:51Z")
I already saw it. I tried using it, but got stuck with other things such as metric, preprocessing etc. Given that training will last for a week, there is not much scope to make errors. ---
**Author:** ![prajjwal1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prajjwal1/32/2246_2.png) [@prajjwal1](https://discuss.huggingface.co/u/prajjwal1)\ **Post date:** [September 25, 2020, 2:43pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/19 "2020-09-25T14:43:24Z")
Also, is there some study or has anyone experimented what happens if we solely rely on MLM and no NSP. How much difference will that make ? RoBERTa showed that NSP didn’t prove to be useful. In this case, does involving NSP help with MLM ? ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [September 25, 2020, 2:51pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/20 "2020-09-25T14:51:37Z")
Well as you found, RoBERTa showed that leaving out NSP yields better results on downstream tasks. Albert then re-added a similar (yet very different) task, namely sentence _order_ prediction, which improved performance on downstream tasks. PS: please don’t post multiple consecutive posts but rather edit your posts to add more information. It’s a bit annoying with the notifications. 🙂 ---
**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](https://discuss.huggingface.co/u/vblagoje)\ **Post date:** [September 25, 2020, 3:39pm UTC](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245/21 "2020-09-25T15:39:26Z")
Quentin, I am not sure dataset itself should cut articles into sentences (unless there is an option for both articles/sentences). Perhaps other models might need entire articles as input. If needed, users can sentence tokenize articles using nltk/spacy and such. I’ll play with the wikipedia dataset in the coming days and I’ll report back to you my experiences. Also, while looking at the dataset I found references to Categories and such. Perhaps equally important objective for wikipedia dateset is to keep it as clean as possible. [Next page](https://discuss.huggingface.co/t/pre-train-bert-from-scratch/1245.md?page=2)