# Loading SentencePiece tokenizer

**URL:** <https://discuss.huggingface.co/t/loading-sentencepiece-tokenizer/20814>\
**Category:** Beginners\
**Created:** [July 25, 2022, 7:42am UTC](https://discuss.huggingface.co/t/loading-sentencepiece-tokenizer/20814 "2022-07-25T07:42:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mmukh](https://avatars.discourse-cdn.com/v4/letter/m/258eb7/32.png) [@mmukh](https://discuss.huggingface.co/u/mmukh)\
**Post date:** [July 25, 2022, 7:42am UTC](https://discuss.huggingface.co/t/loading-sentencepiece-tokenizer/20814/1 "2022-07-25T07:42:23Z")

</div>

When I use SentencePieceTrainer.train(), it returns a .model and .vocab file. However when trying to load it using AutoTokenizer.from\_pretrained() it expects a .json file. How would I get a .json file from the .model and .vocab file?

---

<div class="post-metadata">

**Author:** ![mapama247](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mapama247/32/33424_2.png) [@mapama247](https://discuss.huggingface.co/u/mapama247)\
**Post date:** [July 26, 2022, 9:09am UTC](https://discuss.huggingface.co/t/loading-sentencepiece-tokenizer/20814/2 "2022-07-26T09:09:45Z")

</div>

Did you save the model first? You can do that using the [save\_pretrained()](https://huggingface.co/docs/transformers/v4.20.1/en/main_classes/configuration#transformers.PretrainedConfig.save_pretrained) function, and then simply load the tokenizer by providing the model’s directory (where all the necessary files have been stored) to the [from\_pretrained()](https://huggingface.co/docs/transformers/v4.20.1/en/main_classes/configuration#transformers.PretrainedConfig.from_pretrained) function.

---

<div class="post-metadata">

**Author:** ![StephennFernandes](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stephennfernandes/32/17230_2.png) [@StephennFernandes](https://discuss.huggingface.co/u/StephennFernandes)\
**Post date:** [October 22, 2023, 4:51pm UTC](https://discuss.huggingface.co/t/loading-sentencepiece-tokenizer/20814/3 "2023-10-22T16:51:36Z")

</div>

@mapama247 i am actully stuck here.

i tried loading the sentencepiece trained tokenizer using the following script

```auto
tok=tokenizers.SentencePieceUnigramTokenizer.from_spm("tokenizer.model")
tok.save_pretrained("hf_format_tokenizer")

```

I get the following error:

```auto
AttributeError: 'SentencePieceUnigramTokenizer' object has no attribute 'save_pretrained'

```

---

<div class="post-metadata">

**Author:** ![mapama247](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mapama247/32/33424_2.png) [@mapama247](https://discuss.huggingface.co/u/mapama247)\
**Post date:** [October 24, 2023, 12:49pm UTC](https://discuss.huggingface.co/t/loading-sentencepiece-tokenizer/20814/4 "2023-10-24T12:49:07Z")

</div>

Hi @StephennFernandes,  
This is because in the previous message I was talking about the AutoTokenizer class from transformers. I cannot reproduce your problem because you’re loading from local, but I see that the SentencePieceUnigramTokenizer that you’re using has a couple methods that you could try: `save` and `save_model`. You can see them doing `dir(tokenizers.SentencePieceUnigramTokenizer)`