# Further Pretrain Basic BERT for sequence classification **URL:** \ **Category:** 🤗Transformers\ **Created:** [October 9, 2020, 12:30am UTC](https://discuss.huggingface.co/t/further-pretrain-basic-bert-for-sequence-classification/1436 "2020-10-09T00:30:19Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![TonyGong97](https://avatars.discourse-cdn.com/v4/letter/t/b487fb/32.png) [@TonyGong97](https://discuss.huggingface.co/u/TonyGong97)\ **Post date:** [October 9, 2020, 12:30am UTC](https://discuss.huggingface.co/t/further-pretrain-basic-bert-for-sequence-classification/1436/1 "2020-10-09T00:30:19Z")
Has anyone successfully used Hugging Face to further pretrain Baisc BERT model and apply it to sequence classification task ? I tried to find any relevant posts/documents on Hugging Face website but failed to find any useful information. The usual way to further pretrain BERT is to use original google BERT implementation. I want to stick with Huggingface and see if there is a way to work around with this further pretraining task. If anyone could offer some help or guidances I sincerely appreciate it. ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [October 9, 2020, 9:14am UTC](https://discuss.huggingface.co/t/further-pretrain-basic-bert-for-sequence-classification/1436/2 "2020-10-09T09:14:48Z")
You seem to be looking for the term _finetuning_. You should be able to find a lot of information if you search for that term. Here is a good one to get you started: [https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers\_multi\_label\_classification.ipynb](https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_multi_label_classification.ipynb) ---
**Author:** ![TonyGong97](https://avatars.discourse-cdn.com/v4/letter/t/b487fb/32.png) [@TonyGong97](https://discuss.huggingface.co/u/TonyGong97)\ **Post date:** [October 9, 2020, 2:35pm UTC](https://discuss.huggingface.co/t/further-pretrain-basic-bert-for-sequence-classification/1436/3 "2020-10-09T14:35:49Z")
Hi Bram. Thank you very much for your kind reply. After seeing your reply I rethink of the so called further pre-training that I want to do. I realize that I can actually just directly train, for example, **BertForSequenceClassification** model and by fitting the model to my data I am actually already doing the further pre-training and classification at the same time, because all the model parameters will be updated according to my new data(If I don’t intentionally freeze the weight update for all the transformer layer). Am I understanding this correctly ? ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [October 9, 2020, 4:05pm UTC](https://discuss.huggingface.co/t/further-pretrain-basic-bert-for-sequence-classification/1436/4 "2020-10-09T16:05:18Z")
That is correct but only if you use `BertForSequenceClassification.from_pretrained` to load the existing weights. You can then use that model to further train the model. If you do not use `from_pretrained`, you will train a new model from-scratch and that is probably not what you want. This process is called finetuning the model on a downstream task. Do not call it pretrain, that is confusing. Pretraining is the initial training that was done to get the weights that currently exist in the models that we can use. ---
**Author:** ![TonyGong97](https://avatars.discourse-cdn.com/v4/letter/t/b487fb/32.png) [@TonyGong97](https://discuss.huggingface.co/u/TonyGong97)\ **Post date:** [October 9, 2020, 5:55pm UTC](https://discuss.huggingface.co/t/further-pretrain-basic-bert-for-sequence-classification/1436/5 "2020-10-09T17:55:58Z")
Understood. Thanks so much Bram.This is of great help to me ! 🙂