# How do i take only "BERT" weights from BertForSequenceClassification model?

**URL:** <https://discuss.huggingface.co/t/how-do-i-take-only-bert-weights-from-bertforsequenceclassification-model/14794>\
**Category:** 🤗Transformers\
**Created:** [February 16, 2022, 2:32pm UTC](https://discuss.huggingface.co/t/how-do-i-take-only-bert-weights-from-bertforsequenceclassification-model/14794 "2022-02-16T14:32:59Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![lucadini](https://avatars.discourse-cdn.com/v4/letter/l/d9b06d/32.png) [@lucadini](https://discuss.huggingface.co/u/lucadini)\
**Post date:** [February 16, 2022, 2:32pm UTC](https://discuss.huggingface.co/t/how-do-i-take-only-bert-weights-from-bertforsequenceclassification-model/14794/1 "2022-02-16T14:32:59Z")

</div>

Hi everyone,

I’m currently trying to see if BERT performances get better on a binary classification task, after I did a fine-tuning on another task (regression).

My pipeline is:

1. bert base → finetune on regression → fine tune on classification → test  
vs
2. bert base → finetune on classification → test

For all the steps I’m using run\_glue\_no\_trainer.py (with little modifications) found in transformers/examples/pytorch/text-classification.  
I have a problem with loading the finetuned-with-regression-BERT weights in AutoModelForSequenceClassification. Looking on github issues I found the parameter “ignore\_mismatched\_sizes=True” that allows me to load weights without error, but then I get this runtime error when training:

huggingface RuntimeError: The size of tensor a (2) must match the size of tensor b (8) at non-singleton dimension 1

I assume that the problem is the parameter ignore\_mismatched\_sizes is for going from a classification head to another classification head with different classes, and so it loads the weights anyway (but the regression output for a sample is of size 1, and for binary classification is of size 2).  
Given the fact I’m not interested in keeping my regression head, how can I take only “BERT” weights of my finetuned model on regression?

I tried:

config = BertConfig.from\_pretrained(BERT\_BASE)  
model = BertModel.from\_pretrained(REGRESSION-FINE-TUNED-BERT, config=config)  
model.save\_pretrained(…)

Using this model for initializing BertForSequenceClassification I don’t get errors but I get this warning I wasn’t expecting since I should have removed the regression/classification head:

Some weights of BertForSequenceClassification were not initialized from the model checkpoint at /home/irene/similarity\_eval/preprocessing/bert\_similarity and are newly initialized: [‘classifier.weight’, ‘classifier.bias’]

Any advice?

Thanks in advance!!