# Convert tensorflow tokenclassifier checkpoint to pytorch

**URL:** <https://discuss.huggingface.co/t/convert-tensorflow-tokenclassifier-checkpoint-to-pytorch/13288>\
**Category:** 🤗Transformers\
**Created:** [January 1, 2022, 1:16pm UTC](https://discuss.huggingface.co/t/convert-tensorflow-tokenclassifier-checkpoint-to-pytorch/13288 "2022-01-01T13:16:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Maaly](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@Maaly](https://discuss.huggingface.co/u/Maaly)\
**Post date:** [January 1, 2022, 1:16pm UTC](https://discuss.huggingface.co/t/convert-tensorflow-tokenclassifier-checkpoint-to-pytorch/13288/1 "2022-01-01T13:16:15Z")

</div>

Hi,

Is there a way to convert the checkpoint of a Bert model that was fine-tuned for token classification task (NER) using tensorflow Bert original script?

I am trying to use “transformer-cli convert” but it doesn’t work except on pretrained models.

I want to share my fine-tuned models on models hub but it can’t be converted.

Thanks,  
Maaly

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [January 1, 2022, 7:02pm UTC](https://discuss.huggingface.co/t/convert-tensorflow-tokenclassifier-checkpoint-to-pytorch/13288/2 "2022-01-01T19:02:33Z")

</div>

Hi,

In HuggingFace, any model that has both an implementation in PyTorch and Tensorflow can be easily converted as follows:

```auto
from transformers import BertForTokenClassification

model = BertForTokenClassification.from_pretrained("name_of_your_directory", from_tf=True)

```

---

<div class="post-metadata">

**Author:** ![Maaly](https://avatars.discourse-cdn.com/v4/letter/m/5f9b8f/32.png) [@Maaly](https://discuss.huggingface.co/u/Maaly)\
**Post date:** [January 2, 2022, 5:51pm UTC](https://discuss.huggingface.co/t/convert-tensorflow-tokenclassifier-checkpoint-to-pytorch/13288/3 "2022-01-02T17:51:06Z")

</div>

Hi,

Thank you for your reply. I tried this approach and got this error:

- 
  1. OSError: Error no file named [‘pytorch\_model.bin’, ‘tf\_model.h5’, ‘model.ckpt.index’, ‘flax\_model.msgpack’] found in directory hub\_models/ena\_all\_9010/anatomy/4e-05-90 or `from_tf` and `from_flax` set to False.

I was thinking of a way to convert the checkpoints of tensorflow fine-tuned model to pytorch as I did with the checkpoints for pretrained Bert models. Here are the files of one Bert fine-tuned model:

- checkpoint
- config.json
- eval
- eval\_results.txt
- eval.tf\_record
- graph.pbtxt
- label2id.pkl
- label\_test.txt
- model.ckpt-3000.data-00000-of-00001
- model.ckpt-3000.index
- model.ckpt-3000.meta
- model.ckpt-4000.data-00000-of-00001
- model.ckpt-4000.index
- model.ckpt-4000.meta
- model.ckpt-5000.data-00000-of-00001
- model.ckpt-5000.index
- model.ckpt-5000.meta
- model.ckpt-6000.data-00000-of-00001
- model.ckpt-6000.index
- model.ckpt-6000.meta
- model.ckpt-6316.data-00000-of-00001
- model.ckpt-6316.index
- model.ckpt-6316.meta
- NER\_result\_conll.txt
- predict.tf\_record
- token\_test.txt
- train.tf\_record

I tried to convert the checkpoints using convert\_bert\_original\_tf\_checkpoint\_to\_pytorch.py but I got the following error:

- raise ModuleAttributeError("’{}’ object has no attribute ‘{}’".format(  
torch.nn.modules.module.ModuleAttributeError: ‘BertForPreTraining’ object has no attribute ‘bias’

I tried to fix it by replacing line 33 with model = BertForTokenClassification(config), then I got the following error:

- transformers/src/transformers/models/bert/modeling\_bert.py", line 156, in load\_tf\_weights\_in\_bert
- if pointer.shape != array.shape:
- AttributeError: ‘NoneType’ object has no attribute ‘shape’

investigating this issue more showed that pointer.shape != array.shape. Infact pointer = None for the following layers:  
[‘bert’, ‘pooler’, ‘dense’, ‘bias’] , where array.shape = (768,)  
[‘bert’, ‘pooler’, ‘dense’, ‘kernel’], where array.shape = (768, 768)  
[‘output\_bias’], where array.shape = (7,)  
[‘output\_weights’], where array.shape = (7, 768)

I’d appreciate if someone can help me in figuring out how to fix the pointer for those layers.

Thanks