# Loading pretrained SentencePiece tokenizer from Fairseq

**URL:** <https://discuss.huggingface.co/t/loading-pretrained-sentencepiece-tokenizer-from-fairseq/1326>\
**Category:** 🤗Tokenizers\
**Created:** [October 1, 2020, 7:15am UTC](https://discuss.huggingface.co/t/loading-pretrained-sentencepiece-tokenizer-from-fairseq/1326 "2020-10-01T07:15:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![proxyht](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@proxyht](https://discuss.huggingface.co/u/proxyht)\
**Post date:** [October 1, 2020, 7:15am UTC](https://discuss.huggingface.co/t/loading-pretrained-sentencepiece-tokenizer-from-fairseq/1326/1 "2020-10-01T07:15:33Z")

</div>

Hello. I have a pretrained RoBERTa model on fairseq, which contains `dict.txt`, `model.pt`, `sentencepiece.bpe.model`.

I have found a way to convert a fairseq checkpoint to huggingface format in [https://github.com/huggingface/transformers/blob/master/src/transformers/convert\_roberta\_original\_pytorch\_checkpoint\_to\_pytorch.py](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py)

Howerver, I couldn’t find a similar method to convert the tokenizer in fairseq `sentencepiece.bpe.model` to huggingface’s format.  
Is there any existing solution? Or do I have to convert it by myself?

Thanks.

---

<div class="post-metadata">

**Author:** ![gurvinder](https://avatars.discourse-cdn.com/v4/letter/g/f08c70/32.png) [@gurvinder](https://discuss.huggingface.co/u/gurvinder)\
**Post date:** [October 7, 2020, 9:01am UTC](https://discuss.huggingface.co/t/loading-pretrained-sentencepiece-tokenizer-from-fairseq/1326/2 "2020-10-07T09:01:02Z")

</div>

@proxyht were you able to convert sentencepiece model to huggingface tokenizer. As I am facing similar issues as well.

---

<div class="post-metadata">

**Author:** ![proxyht](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@proxyht](https://discuss.huggingface.co/u/proxyht)\
**Post date:** [October 7, 2020, 4:16pm UTC](https://discuss.huggingface.co/t/loading-pretrained-sentencepiece-tokenizer-from-fairseq/1326/3 "2020-10-07T16:16:57Z")

</div>

A colleague of mine has figured out a way to work around this issue. Although both Huggingface and Fairseq use spm from google, the tokenizer in Fairseq map the id from spm to the token id in the **dict.txt** file, while Huggingface’s does not

We will have to write a custom Tokenizer in Huggingface to simulate the behavior as in Fairseq

---

<div class="post-metadata">

**Author:** ![gurvinder](https://avatars.discourse-cdn.com/v4/letter/g/f08c70/32.png) [@gurvinder](https://discuss.huggingface.co/u/gurvinder)\
**Post date:** [October 7, 2020, 8:12pm UTC](https://discuss.huggingface.co/t/loading-pretrained-sentencepiece-tokenizer-from-fairseq/1326/4 "2020-10-07T20:12:34Z")

</div>

Thanks. I will look into creating the custom tokenizer then using sentence piece model and tokenize data before passing it to training.

---

<div class="post-metadata">

**Author:** ![gurvinder](https://avatars.discourse-cdn.com/v4/letter/g/f08c70/32.png) [@gurvinder](https://discuss.huggingface.co/u/gurvinder)\
**Post date:** [October 9, 2020, 6:52pm UTC](https://discuss.huggingface.co/t/loading-pretrained-sentencepiece-tokenizer-from-fairseq/1326/5 "2020-10-09T18:52:18Z")

</div>

@proxyht seems like new version released today added support for loading sentenncepiece model. Details in this PR [https://github.com/huggingface/tokenizers/pull/292](https://github.com/huggingface/tokenizers/pull/292) Haven’t tested myself yet. Plan to test in a week or so. Let me know how it goes if you test before.

---

<div class="post-metadata">

**Author:** ![gurvinder](https://avatars.discourse-cdn.com/v4/letter/g/f08c70/32.png) [@gurvinder](https://discuss.huggingface.co/u/gurvinder)\
**Post date:** [October 21, 2020, 12:55pm UTC](https://discuss.huggingface.co/t/loading-pretrained-sentencepiece-tokenizer-from-fairseq/1326/6 "2020-10-21T12:55:38Z")

</div>

For reference in tokenizer version 0.9.2 there is a function to load spm model. You need to run these two commands before to install the dependencies

> pip install protobuf  
> wget [https://raw.githubusercontent.com/google/sentencepiece/master/python/sentencepiece\_model\_pb2.py](https://raw.githubusercontent.com/google/sentencepiece/master/python/sentencepiece_model_pb2.py)

then you can instantiate huggingface tokenizer using pretrain sentencepiece model as

`tok = tokenizers.SentencePieceUnigramTokenizer.from_spm("spm.model")`