# How to load a google's bert ckpt using tf2 **URL:** \ **Category:** 🤗Transformers\ **Created:** [August 5, 2020, 9:37am UTC](https://discuss.huggingface.co/t/how-to-load-a-googles-bert-ckpt-using-tf2/600 "2020-08-05T09:37:11Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![vicky](https://avatars.discourse-cdn.com/v4/letter/v/b19c9b/32.png) [@vicky](https://discuss.huggingface.co/u/vicky)\ **Post date:** [August 5, 2020, 9:37am UTC](https://discuss.huggingface.co/t/how-to-load-a-googles-bert-ckpt-using-tf2/600/1 "2020-08-05T09:37:11Z")
Is there any way to load a google released ckpt in [https://github.com/google-research/bert](https://github.com/google-research/bert) (because some settings are not provided in hf transformers) using tf 2+ ? I’ve found docs on loading those models and converting them to pytorch models in here: [https://huggingface.co/transformers/converting\_tensorflow\_models.html](https://huggingface.co/transformers/converting_tensorflow_models.html) However, it only works with pytorch. ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [August 6, 2020, 5:15pm UTC](https://discuss.huggingface.co/t/how-to-load-a-googles-bert-ckpt-using-tf2/600/2 "2020-08-06T17:15:57Z")
Have you looked at using the keras-bert library (instead of using huggingface transformers library)? > **[GitHub - CyberZHG/keras-bert: Implementation of BERT that could load official...](https://github.com/CyberZHG/keras-bert)** > > Implementation of BERT that could load official pre-trained models for feature extraction and prediction - GitHub - CyberZHG/keras-bert: Implementation of BERT that could load official pre-trained ... keras-bert works with TF2. I used it to load a bert model from the tensorflow hub into colab. I copied it onto my google drive first, and loaded from that copy, but I believe it is also possible to load directly using !wget -q [https://storage.googleapis.com/bert\_models/2018\_10\_18/uncased\_L-12\_H-768\_A-12.zip](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip) !unzip -o uncased\_L-12\_H-768\_A-12.zip So far as I can tell, models loaded with keras-bert are not compatible with transformers. I tried to get my fine-tuned model into transformers format (because I wanted to use Jesse Vig’s visualisations), but I never got it to work. I believe there is something about the way the transformers code is written that means you can’t use non-transformers models in place of transformers models, or at least not without some serious poking of the python. However, I could be mistaken. ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [August 7, 2020, 10:19am UTC](https://discuss.huggingface.co/t/how-to-load-a-googles-bert-ckpt-using-tf2/600/3 "2020-08-07T10:19:40Z")
Hello again, it looks like I was mistaken. The suggestion to use keras-bert instead still stands, but I’ve had another look at the transformers documentation and it certainly looks like it should be possible to use the google models within transformers. See eg this issue [https://github.com/huggingface/transformers/issues/3070](https://github.com/huggingface/transformers/issues/3070). Are you explicitly loading the google model’s config file? What happens if you try using the first part of the pytorch conversion script to read the google model in? ---
**Author:** ![vicky](https://avatars.discourse-cdn.com/v4/letter/v/b19c9b/32.png) [@vicky](https://discuss.huggingface.co/u/vicky)\ **Post date:** [August 14, 2020, 11:41am UTC](https://discuss.huggingface.co/t/how-to-load-a-googles-bert-ckpt-using-tf2/600/4 "2020-08-14T11:41:17Z")
Thanks a lot for your comments! Yes I’m explicitly loading the config file, which I believe is almost the same in both tf and pytorch. I’ve tried to use the pytorch conversion script to load the model in pytorch and then convert the pytorch model to tensorflow model, but I can’t find any scripts to convert the pytorch model to tensorflow one… However, I’ve found a workaround, which is a temper fix: Since the may problem is the mismatch of tensor names, I load the google checkpoint file and rename tensors to match those in transformers. Fortunately the mismatches are easily located and only need sightly modification.