# XLNetForSqeuenceClassification warnings

**URL:** <https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096>\
**Category:** 🤗Transformers\
**Created:** [September 12, 2020, 11:43am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096 "2020-09-12T11:43:01Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\
**Post date:** [September 12, 2020, 11:43am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/1 "2020-09-12T11:43:01Z")

</div>

Hi,

In Google Colab notebook, I install (!pip transformers) and import XLNetForSequenceClassification model. When I instantiate the model the firs time (before training), I get the below:

Some weights of the model checkpoint at xlnet-base-cased were not used when initializing XLNetForSequenceClassification: [‘lm\_loss.weight’, ‘lm\_loss.bias’]

- This IS expected if you are initializing XLNetForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing XLNetForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).  
Some weights of XLNetForSequenceClassification were not initialized from the model checkpoint at xlnet-base-cased and are newly initialized: [‘sequence\_summary.summary.weight’, ‘sequence\_summary.summary.bias’, ‘logits\_proj.weight’, ‘logits\_proj.bias’]  
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

After training, I save the model’s state\_dict using torch.save(). When I load this model for inference (using torch.load), I get the same messages as above.

Why would I get the messages post training?

---

<div class="post-metadata">

**Author:** ![xin](https://avatars.discourse-cdn.com/v4/letter/x/bbce88/32.png) [@xin](https://discuss.huggingface.co/u/xin)\
**Post date:** [September 23, 2020, 8:09am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/2 "2020-09-23T08:09:49Z")

</div>

I got the exact same problem for XLNet’s QuestionsAnsweringModel when I use the pretrained xlnet-base-cased model. It was working well probably 2 months ago and I didn’t change anything.

When I ignore this warning, the training loss will be too huge in each epoch.

I want to know is there a solution or reason now?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [September 23, 2020, 8:20am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/3 "2020-09-23T08:20:45Z")

</div>

@Karthik12 @xin Please post the code that you use to: 1. load the model before training 2. the code that you use to save the model after training 3. the code that you use to load the saved model.

---

<div class="post-metadata">

**Author:** ![xin](https://avatars.discourse-cdn.com/v4/letter/x/bbce88/32.png) [@xin](https://discuss.huggingface.co/u/xin)\
**Post date:** [September 23, 2020, 8:47am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/4 "2020-09-23T08:47:21Z")

</div>

1. load the model before training:

Then, I got the following warning:

> Some weights of the model checkpoint at xlnet-base-cased were not used when initializing XLNetForQuestionAnswering: [‘lm\_loss.weight’, ‘lm\_loss.bias’]
> 
> - This IS expected if you are initializing XLNetForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
> - This IS NOT expected if you are initializing XLNetForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).  
> Some weights of XLNetForQuestionAnswering were not initialized from the model checkpoint at xlnet-base-cased and are newly initialized: [‘start\_logits.dense.weight’, ‘start\_logits.dense.bias’, ‘end\_logits.dense\_0.weight’, ‘end\_logits.dense\_0.bias’, ‘end\_logits.LayerNorm.weight’, ‘end\_logits.LayerNorm.bias’, ‘end\_logits.dense\_1.weight’, ‘end\_logits.dense\_1.bias’, ‘answer\_class.dense\_0.weight’, ‘answer\_class.dense\_0.bias’, ‘answer\_class.dense\_1.weight’]  
> You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

1. the code that you use to save the model after training:

model.train\_model(train\_data, output\_dir=“/media/bizon/DATA/QA/model\_saved”)

1. the code that you use to load the saved model:  
It is from my previous working code, due to the current XLNet doesn’t work.

model = QuestionAnsweringModel(‘xlnet’, ‘/content/drive/MyDrive/transformers/apex/outputs/checkpoint-24435-epoch-3’)

=================================  
The environment I am using:

Ubuntu 18.04.3 LTS

The libraries I used are:

```
Name: transformers
Version: 3.1.0

Name: simpletransformers
Version: 0.48.3

Name: torch
Version: 1.5.0

```

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [September 23, 2020, 9:35am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/5 "2020-09-23T09:35:09Z")

</div>

Sorry, I am not familiar with simpletransformers so I cannot help you further. I do not have the time to dig into how that library works.

---

<div class="post-metadata">

**Author:** ![xin](https://avatars.discourse-cdn.com/v4/letter/x/bbce88/32.png) [@xin](https://discuss.huggingface.co/u/xin)\
**Post date:** [September 23, 2020, 9:54am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/6 "2020-09-23T09:54:51Z")

</div>

Thank you anyway. I guess it is not because of simpletransformers.

Hope @Karthik12 could give code if he didn’t use simpletransformers but still had the same warning.

---

<div class="post-metadata">

**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\
**Post date:** [September 23, 2020, 11:16am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/7 "2020-09-23T11:16:19Z")

</div>

I retried with BertForSequenceClassification and I got the somewhat similar message I posted initially:

#BertClass

class BertClassification(torch.nn.Module):  
def **init** (self, num\_labels=1):  
super(BertClassification, self). **init** ()  
self.num\_labels = num\_labels  
self.bert = BertForSequenceClassification.from\_pretrained(‘bert-base-uncased’, num\_labels = self.num\_labels, output\_attentions = False,output\_hidden\_states = False)

def forward(self, input\_ids, token\_type\_ids=None, attention\_mask=None, labels=None):

```
if labels is None:
  logits = self.bert(input_ids=input_ids, attention_mask=attention_mask,token_type_ids=token_type_ids, labels = None)
  return logits
else:
  loss, logits = self.bert(input_ids=input_ids, attention_mask=attention_mask,token_type_ids=token_type_ids, labels = labels)
  return loss, logits

```

* * *

#Create Data to process for Bert  
train\_sentences = X\_train[‘text’].values  
train\_sentences = [sentence for sentence in train\_sentences]  
train\_labels = Y\_train[‘label’].values

#Tokenize the texts  
tokenizer = BertTokenizer.from\_pretrained(‘bert-base-uncased’, do\_lower\_case=True)

for sent in train\_sentences:  
encoded\_sent = tokenizer.encode\_plus(  
text=sent,  
add\_special\_tokens=True,  
max\_length=max\_len,  
padding=‘max\_length’,  
return\_attention\_mask=True,  
truncation=True  
)  
#Add the outputs to the lists  
input\_ids.append(encoded\_sent.get(‘input\_ids’))  
attention\_masks.append(encoded\_sent.get(‘attention\_mask’))

#Convert lists to tensors  
train\_inputs = torch.tensor(input\_ids)  
train\_masks = torch.tensor(attention\_masks)  
train\_labels = torch.tensor(train\_labels, dtype=torch.long, device=device)

#Create Iterators for Train and Valid  
train\_data = TensorDataset(train\_inputs, train\_masks, train\_labels)  
train\_sampler = RandomSampler(train\_data)  
train\_dataloader = DataLoader(train\_data, sampler=train\_sampler, batch\_size=batch\_size)

model = BertClassification(num\_labels=2)  
optimizer = AdamW(model.parameters(), lr = 1e-5, eps = 1e-8)  
total\_steps = len(train\_dataloader) \* num\_epochs  
scheduler = get\_linear\_schedule\_with\_warmup(optimizer, num\_warmup\_steps = 0, num\_training\_steps = total\_steps)

#Train Model  
model = train(model=model, num\_epochs=num\_epochs,optimizer=optimizer,scheduler=scheduler, train\_dataloader=train\_dataloader, valid\_dataloader=validation\_dataloader)

#Save Model  
model\_save = model.module if hasattr(model, ‘module’) else model  
checkpoint = {‘epochs’: epochs, ‘state\_dict’: model\_save.state\_dict() }  
torch.save(checkpoint, save\_path)

* * *

#Load Model for inference:  
model = load\_model(model\_save\_path)

Colab Mesasge:

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: [‘cls.predictions.bias’, ‘cls.predictions.transform.dense.weight’, ‘cls.predictions.transform.dense.bias’, ‘cls.predictions.decoder.weight’, ‘cls.seq\_relationship.weight’, ‘cls.seq\_relationship.bias’, ‘cls.predictions.transform.LayerNorm.weight’, ‘cls.predictions.transform.LayerNorm.bias’]

- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).  
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: [‘classifier.weight’, ‘classifier.bias’]  
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [September 23, 2020, 11:51am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/8 "2020-09-23T11:51:14Z")

</div>

Can you post the code for `load_model`? You should probably use

```python
model.save_pretrained(save_dir)
model = BertClassification.from_pretrained(save_dir)

```

where `BertClassification` subclasses `transformers`’ `PreTrainedModel`.

---

<div class="post-metadata">

**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\
**Post date:** [September 23, 2020, 11:55am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/9 "2020-09-23T11:55:52Z")

</div>

load\_model function does this:

checkpoint = torch.load(save\_path)  
model\_state\_dict = checkpoint[‘state\_dict’]  
model = BertClassification(num\_labels=num\_labels)  
model.load\_state\_dict(model\_state\_dict)

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [September 23, 2020, 12:34pm UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/10 "2020-09-23T12:34:02Z")

</div>

Okay, so after some digging:

- This is expected behqviour
- When you run `load_model` you re-load the original pretrained `bert-base-uncased` because you reinitialize the model. This line is run again: `BertForSequenceClassification.from_pretrained(‘bert-base-uncased’`
- The message that you get is because “bert-base-uncased” contains the weights for BertForPretraining, which has the following submodules:

> <https://github.com/huggingface/transformers/blob/28cf873036d078b47fb9dd38ac3421a7c874da44/src/transformers/modeling_bert.py#L858-L865>

The warning that you get tells you that the original model has weights (heads) that `BertForSequenceClassification` does not have. The final classifier layer in BertForSequenceClassification is randomly initialized and does not get the weights from the pretrained modelùs `seq_relationship`. As said, this warning will return every time you use `BertForSequenceClassification.from_pretrained("bert-base-uncased")`.

The correct way, and most pythonic, is to subclass the PreTrainedModel and use the save\_pretrained and load\_pretrained methods directly.

---

<div class="post-metadata">

**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\
**Post date:** [September 23, 2020, 12:42pm UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/11 "2020-09-23T12:42:08Z")

</div>

Thank you @BramVanroy. Few things:

1. Subclass the PreTrainedModel - how can I do this as I assume it is not the same as below?  
BertSequenceClassification = BertForSequenceClassification.from\_pretrained(‘bert-base-uncased’)

2. Use the save\_pretrained: I will try the below:  
model.save\_pretrained(str)

3. Load\_pretrained methods directly: I will try the below:  
model.from\_pretrained(str)

---

<div class="post-metadata">

**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\
**Post date:** [September 23, 2020, 1:05pm UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/12 "2020-09-23T13:05:14Z")

</div>

Ok, I think I got what you said.

I will subclass the class "PreTrainedModel " and override its methods (save\_pretrained and load\_pretrained) to change the behavior in these methods. Let me try.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [September 23, 2020, 1:22pm UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/13 "2020-09-23T13:22:56Z")

</div>

No, you do not need to override those methods. They should work without needing to change anything. However, now that I take a better look at your code, why do you need a separate model? Why don’t you just use the BertForSequenceClassification model itself? You do not add any layers, right? So I think you can use this

```python
num_labels = 1
config = AutoConfig.from_pretrained("bert-base-uncased",
                                    num_labels=num_labels,
                                    output_attentions=False,
                                    output_hidden_states=False)
bert = BertForSequenceClassification.from_pretrained("bert-base-uncased", config=config)

# train model here...

# Saving/loading using built-in functionality
bert.save_pretrained(save_dir)
# Load the correct weights directly
bert = BertForSequenceClassification.from_pretrained(save_dir,
                                                     num_labels=num_labels,
                                                     output_attentions=False,
                                                     output_hidden_states=False)

# ...or using your own save/load method
checkpoint = {"epochs": epochs, "state_dict": model_save.state_dict()}
torch.save(checkpoint, save_path)

checkpoint = torch.load(save_path)
# NO from_pretrained so we don't unnecessarily load weights twice"
bert = BertForSequenceClassification(config)
bert.load_state_dict(checkpoint["state_dict"])

```

---

<div class="post-metadata">

**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\
**Post date:** [September 23, 2020, 1:30pm UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/14 "2020-09-23T13:30:17Z")

</div>

The fact I instantiate the model (again) in load\_model function gives me the messages as you said. True,as I am not adding extra layers, I could do the above.

Thank you, this helps to understand the approach.

---

<div class="post-metadata">

**Author:** ![david-waterworth](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@david-waterworth](https://discuss.huggingface.co/u/david-waterworth)\
**Post date:** [December 23, 2020, 12:05am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/16 "2020-12-23T00:05:26Z")

</div>

@BramVanroy is this expected? It confused me as well. I trained a model by adapting `run_mlm.py` and then fine-tuned using another script I adapted from `run_glue.py` (these scripts are generic, they use Auto\*)

If I load the fine-tuned model using `AutoModel.from_pretrained()` it returns the model as a `RobertaModel` which produces the warning, Using `AutoModelForSequenceClassification.from_pretrained()` returns the same model as a `RobertaModelForSequenceClassification`

Since my model config contains

“architectures”: [  
“RobertaForSequenceClassification”  
],

I originally expected that `AutoModel` would return this class but I guess not.

---

<div class="post-metadata">

**Author:** ![facehugger2020](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/facehugger2020/32/3052_2.png) [@facehugger2020](https://discuss.huggingface.co/u/facehugger2020)\
**Post date:** [December 23, 2020, 12:45am UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/17 "2020-12-23T00:45:51Z")

</div>

Wow. This thread is really complex.

Here’s what I do to serialize my model after training:

```python
torch.save(model.state_dict(), save_model_filename)

```

Here’s what I do to deserialize my model:

```python
model = AutoModel.from_pretrained(model_name)
model.load_state_dict(torch.load(save_model_filename))

```

I’ve use the above code successfully. Let me know if I’ve done anything wrong.

```auto

```

---

<div class="post-metadata">

**Author:** ![PremalMatalia](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/premalmatalia/32/1457_2.png) [@PremalMatalia](https://discuss.huggingface.co/u/PremalMatalia)\
**Post date:** [April 3, 2021, 3:12pm UTC](https://discuss.huggingface.co/t/xlnetforsqeuenceclassification-warnings/1096/18 "2021-04-03T15:12:59Z")

</div>

@Karthik12 …Is this issue solved by solution provided by @BramVanroy ? I am also getting same warning and model is not behaving correctly.

Same question has been asked in below:

> [@How to load weights for Pre-trained model for Question Answering?](https://discuss.huggingface.co/t/how-to-load-weights-for-pre-trained-model-for-question-answering/5287):
>
> What should I change in below snippet to get consistent and accurate output? Code Snippet: from transformers import ElectraTokenizer, TFElectraForQuestionAnswering, ElectraConfig import tensorflow as tf configuration = ElectraConfig() tokenizer = ElectraTokenizer.from\_pretrained(‘google/electra-small-discriminator’) TFElect = TFElectraForQuestionAnswering(configuration) #model = TFElectraForQuestionAnswering.from\_pretrained(‘google/electra-small-discriminator’) model = TFElect.from\_pretr…