# How can I convert a model created with fairseq? **URL:** \ **Category:** Beginners\ **Created:** [August 2, 2020, 7:20pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564 "2020-08-02T19:20:27Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 2, 2020, 7:20pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/1 "2020-08-02T19:20:27Z")
Hi, I fine tuned facebook’s model mbart.cc25 for machine translation with Fairseq, it saved its model as checkpoint\_\*.pt. How can I use it now with Transformers, is it possible? Thanks ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [August 2, 2020, 8:45pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/2 "2020-08-02T20:45:05Z")
Unless the naming conventions that are used in transformers are the same as in fairseq, this is not possible out of the box. However, with a bit of digging you should be able to map them. @sshleifer will know the answer. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [August 3, 2020, 6:02am UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/3 "2020-08-03T06:02:55Z")
maybe [this](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bart_original_pytorch_checkpoint_to_pytorch.py) will help. This file contains utilities for converting fariseq bart checkpoints to HF format. ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 3, 2020, 6:15am UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/4 "2020-08-03T06:15:58Z")
Great. Thanks, I will try it. Probably I have to change little bit since the script is for bart ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 3, 2020, 2:48pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/5 "2020-08-03T14:48:11Z")
which fairseq `arch` are you converting? I would be very excited (and use) a PR that converts other fairseq formats! ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 3, 2020, 4:00pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/6 "2020-08-03T16:00:51Z")
Hi @sshleifer, as mentioned above I fine tuned mbart.cc25 for machine translation (en-de) with Fairseq. It was actually just for learning purpose, but since it was trained for many hours on multiple gpus, I though it would be good also for other if I put it to huggingface’s models zoo if I am able to convert it. ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 3, 2020, 10:42pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/7 "2020-08-03T22:42:04Z")
yes that would be awesome! I just pushed a script to the `convert-mbart` branch, let me know if that works. [pr here](https://github.com/huggingface/transformers/pull/6230) If your config is different than `mbart-large-en-ro`, you should pass it in. 1. Let me know if that works! 2. What was your final BLEU/how long did it take to train? ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 4, 2020, 3:59pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/9 "2020-08-04T15:59:11Z")
Cool! thanks a lot, I will try it. ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 4, 2020, 4:15pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/10 "2020-08-04T16:15:02Z")
Btw, when I run the script I got following error message: ``` Traceback (most recent call last): File "./convert_mbart_original_checkpoint_to_pytorch.py", line 7, in from .convert_bart_original_pytorch_checkpoint_to_pytorch import remove_ignore_keys_ ModuleNotFoundError: No module named ' __main__.convert_bart_original_pytorch_checkpoint_to_pytorch'; ' __main__' is not a package ``` After I change `from .convert_bart_original_pytorch_checkpoint_to_pytorch import remove_ignore_keys_` to `from convert_bart_original_pytorch_checkpoint_to_pytorch import remove_ignore_keys_` (just removing the dot), the script can run ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 10, 2020, 3:04am UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/11 "2020-08-10T03:04:13Z")
> [@cahya](#): > > After I change `from .convert_bart_original_pytorch_checkpoint_to_pytorch import remove_ignore_keys_` to `from convert_bart_original_pytorch_checkpoint_to_pytorch import remove_ignore_keys_` (just removing the dot), the script can run Let us know when you have succeeded! I’m excited about this! ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 10, 2020, 3:23pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/12 "2020-08-10T15:23:20Z")
Hi, sorry for the late answer. I tried to convert it with this command ```auto python ~/Work/transformers/src/transformers/convert_mbart_original_checkpoint_to_pytorch.py --hf_config facebook/mbart-large-cc25 wmt14-en-de/checkpoint_best.pt out ``` but I get following error message: ```auto Traceback (most recent call last): File "../../transformers/src/transformers/convert_mbart_original_checkpoint_to_pytorch.py", line 36, in model = convert_fairseq_mbart_checkpoint_from_disk(args.fairseq_path, hf_config_path=args.hf_config) File "../../transformers/src/transformers/convert_mbart_original_checkpoint_to_pytorch.py", line 18, in convert_fairseq_mbart_checkpoint_from_disk model.model.load_state_dict(state_dict) File "/home/wirawan/miniconda3/envs/transformers-cuda9/lib/python3.7/site-packages/torch/nn/modules/module.py", line 847, in load_state_dict self. __class__. __name__ , "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for BartModel: Unexpected key(s) in state_dict: "decoder.output_projection.weight". ``` However it did work to convert the original mbart-finetuned\_en-ro model from facebook [https://dl.fbaipublicfiles.com/fairseq/models/mbart/mbart.cc25.ft.enro.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/mbart/mbart.cc25.ft.enro.tar.gz) There was a bug report with the same error message [https://github.com/pytorch/fairseq/issues/2031](https://github.com/pytorch/fairseq/issues/2031) , but I can’t make the solution/work around to work. ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 11, 2020, 3:16pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/13 "2020-08-11T15:16:52Z")
I can add “decoder.output\_projection.weight” to the ignore\_keys in src/transformers/convert\_bart\_original\_pytorch\_checkpoint\_to\_pytorch.py, then the conversion will run without error message and save the model as pytorch\_model.bin, but I guess this is not the correct way to solve it? ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 11, 2020, 7:02pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/14 "2020-08-11T19:02:27Z")
no that is a good way to solve that. Just make sure that the “decoder.output\_projection.weight” param is all zero (it should be). ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 13, 2020, 8:43am UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/15 "2020-08-13T08:43:26Z")
Unfortunately the “decoder.output\_projection.weight” param is not zero at all, as you can see it here: ![mbart-fairseq-01](https://us1.discourse-cdn.com/hellohellohello/original/2X/f/fcb2ad7c80c8c117f098e4fd2e6c814892689d29.jpeg) Probably it is the same as decoder.embed\_tokens.weight acording to [fairseq/transformer.py at 522c76ba1646cd5ec2cd4be29392f53d40aec50a · facebookresearch/fairseq · GitHub](https://github.com/pytorch/fairseq/blob/522c76ba1646cd5ec2cd4be29392f53d40aec50a/fairseq/models/transformer.py#L627) ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 13, 2020, 4:43pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/16 "2020-08-13T16:43:43Z")
Interesting. I’d try renaming that parameter to `model.final_logits_bias` and then checking that your model is still good after conversion. ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 13, 2020, 5:10pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/17 "2020-08-13T17:10:55Z")
Btw, I have a dumb question, how can I translate a text using transformers mbart? I know how to do it with fairseq, but not with transformers 😊 ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [August 15, 2020, 7:41am UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/18 "2020-08-15T07:41:21Z")
Have a look at the seq2seq tutorial: [https://github.com/huggingface/transformers/tree/master/examples/seq2seq](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 15, 2020, 2:21pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/19 "2020-08-15T14:21:57Z")
Thanks, I will try it ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [August 17, 2020, 6:36pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/20 "2020-08-17T18:36:44Z")
@sshleifer For testing purpose I converted the fairseqs mbart to transformers mbart where I ignored the decoder.output\_projection.weight and uploaded the result to huggigface model hub as “cahya/mbart-large-en-de” (for some reason it doesn’t show up in [https://huggingface.co/models](https://huggingface.co/models) but I can use/load it in script as pretrained model). Since I want to know if the converted model works, I created a simple jupyter notebook ([https://gist.github.com/cahya-wirawan/0e3eedbcd78c28602dbc554c447aed2a](https://gist.github.com/cahya-wirawan/0e3eedbcd78c28602dbc554c447aed2a)) to do translation following @BramVanroy suggestion. Before I tested with my converted mbart model, I tested it with “facebook/mbart-large-en-ro”, and the translation seems to work properly, but it doesn’t work if I use my mbart model. Does it means that the converted mbart model is incomplete/corrupt since I ignored the “decoder.output\_projection.weight”? Or did I miss something in the jupyter notebook? Thanks ---
**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\ **Post date:** [August 17, 2020, 6:48pm UTC](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564/21 "2020-08-17T18:48:44Z")
I think you need to convert decoder.output\_projection.weight -\> final\_logits\_bias [Next page](https://discuss.huggingface.co/t/how-can-i-convert-a-model-created-with-fairseq/564.md?page=2)