# Weights of pre-trained BERT model not initialized

**URL:** <https://discuss.huggingface.co/t/weights-of-pre-trained-bert-model-not-initialized/4280>\
**Category:** 🤗Transformers\
**Created:** [March 10, 2021, 10:07am UTC](https://discuss.huggingface.co/t/weights-of-pre-trained-bert-model-not-initialized/4280 "2021-03-10T10:07:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stimmot](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@Stimmot](https://discuss.huggingface.co/u/Stimmot)\
**Post date:** [March 10, 2021, 10:07am UTC](https://discuss.huggingface.co/t/weights-of-pre-trained-bert-model-not-initialized/4280/1 "2021-03-10T10:07:37Z")

</div>

I am using the [Language Interpretability Toolkit](https://github.com/pair-code/lit) (LIT) to load and analyze the ‘bert-base-german-cased’ model that I pre-trained on an NER task with HuggingFace.

However, when I’m starting the LIT script with the path to my pre-trained model passed to it, it fails to initialize the weights and tells me:

```auto
    modeling_utils.py:648] loading weights file bert_remote/examples/token-classification/Data/Models/results_21_03_04_cleaned_annotations/04.03._8_16_5e-5_cleaned_annotations/04-03-2021 (15.22.23)/pytorch_model.bin
    modeling_utils.py:739] Weights of BertForTokenClassification not initialized from pretrained model: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']
    modeling_utils.py:745] Weights from pretrained model not used in BertForTokenClassification: ['bert.embeddings.position_ids']

```

It then simply uses the `bert-base-german-cased` version of BERT, which of course doesn’t have my custom labels and thus fails to predict anything. I think it might have to do with PyTorch or HuggingFace, but I can’t find the error.

If relevant, here is how I load my dataset into CoNLL 2003 format (modification of the dataloader scripts):

```auto
    def __init__ (self):

        # Read ConLL Test Files

        self._examples = []

        data_path = "lit_remote/lit_nlp/examples/datasets/NER_Data"
        with open(os.path.join(data_path, "test.txt"), "r", encoding="utf-8") as f:
            lines = f.readlines()

        for line in lines[:2000]:
            if line != "\n":
                token, label = line.split(" ")
                self._examples.append({
                    'token': token,
                    'label': label,
                })
            else:
                self._examples.append({
                    'token': "\n",
                    'label': "O"
                })

    def spec(self):
        return {
            'token': lit_types.Tokens(),
            'label': lit_types.SequenceTags(align="token"),
        }

```

And this is how I initialize the model and start the LIT server (modification of the `simple_pytorch_demo.py` script):

```auto
    def __init__ (self, model_name_or_path):
        self.tokenizer = transformers.AutoTokenizer.from_pretrained(
            model_name_or_path)
        model_config = transformers.AutoConfig.from_pretrained(
            model_name_or_path,
            num_labels=15, # FIXME CHANGE
            output_hidden_states=True,
            output_attentions=True,
        )
        # This is a just a regular PyTorch model.
        self.model = _from_pretrained(
            transformers.AutoModelForTokenClassification,
            model_name_or_path,
            config=model_config)
        self.model.eval()

## Some omitted snippets here

    def input_spec(self) -> lit_types.Spec:
        return {
            "token": lit_types.Tokens(),
            "label": lit_types.SequenceTags(align="token")
        }

    def output_spec(self) -> lit_types.Spec:
        return {
            "tokens": lit_types.Tokens(),
            "probas": lit_types.MulticlassPreds(parent="label", vocab=self.LABELS),
            "cls_emb": lit_types.Embeddings()

```

Anyone has an idea what the issue could be?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [March 10, 2021, 2:40pm UTC](https://discuss.huggingface.co/t/weights-of-pre-trained-bert-model-not-initialized/4280/2 "2021-03-10T14:40:05Z")

</div>

It’s logical to not have bert pooler weights in token classification (we are not using the pooler in this model), the warning suggests you are not using the last version of 🤗 Transformers but in any case, you can safely ignore it.

---

<div class="post-metadata">

**Author:** ![Stimmot](https://avatars.discourse-cdn.com/v4/letter/s/46a35a/32.png) [@Stimmot](https://discuss.huggingface.co/u/Stimmot)\
**Post date:** [March 11, 2021, 8:15am UTC](https://discuss.huggingface.co/t/weights-of-pre-trained-bert-model-not-initialized/4280/3 "2021-03-11T08:15:52Z")

</div>

Okay, I see - but still I don’t get any predictions from the model. So that these weights are not initialized is not the issue? Does that also apply to the embedding position ids?