# Save\_pretrained() on tokenizer does not generate a tokenizer.json file

**URL:** <https://discuss.huggingface.co/t/save-pretrained-on-tokenizer-does-not-generate-a-tokenizer-json-file/102772>\
**Category:** 🤗Transformers\
**Created:** [August 17, 2024, 4:32pm UTC](https://discuss.huggingface.co/t/save-pretrained-on-tokenizer-does-not-generate-a-tokenizer-json-file/102772 "2024-08-17T16:32:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Akirami](https://avatars.discourse-cdn.com/v4/letter/a/3da27b/32.png) [@Akirami](https://discuss.huggingface.co/u/Akirami)\
**Post date:** [August 17, 2024, 4:32pm UTC](https://discuss.huggingface.co/t/save-pretrained-on-tokenizer-does-not-generate-a-tokenizer-json-file/102772/1 "2024-08-17T16:32:49Z")

</div>

I have created a tokenizer and have merged it with the Llama 2 tokenizer, changing the vocab size from 32k to 38k. I have successfully saved the tokenizer using save\_pretrained() and pushed it to the hub. And I’m also able to pull the tokenizer with AutoTokenizer and use it. But in the hub, I do not see any tokenizer.json file. The .save\_pretrained() only generated a tokenizer\_config.json, tokenizer.model and special\_tokens\_map.json

How do I get the tokenizer.json file?

---

<div class="post-metadata">

**Author:** ![mahmutc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mahmutc/32/52583_2.png) [@mahmutc](https://discuss.huggingface.co/u/mahmutc)\
**Post date:** [August 19, 2024, 10:47am UTC](https://discuss.huggingface.co/t/save-pretrained-on-tokenizer-does-not-generate-a-tokenizer-json-file/102772/2 "2024-08-19T10:47:54Z")

</div>

hi @Akirami  
Can you please share your code snippet? I don’t know if you need something similar to the following but for tokenizer rather than model.

> [@How to create a config.json after saving a model](https://discuss.huggingface.co/t/how-to-create-a-config-json-after-saving-a-model/10459/4):
>
> You need to subclass it to have the save\_pretrained methods available. So instead of class Mean\_Pooling\_Model(nn.Module): use from transformers.modeling\_utils import PreTrainedModel class Mean\_Pooling\_Model(PreTrainedModel): It will add extra functionality on top of nn.Module.

---

<div class="post-metadata">

**Author:** ![Akirami](https://avatars.discourse-cdn.com/v4/letter/a/3da27b/32.png) [@Akirami](https://discuss.huggingface.co/u/Akirami)\
**Post date:** [August 19, 2024, 12:15pm UTC](https://discuss.huggingface.co/t/save-pretrained-on-tokenizer-does-not-generate-a-tokenizer-json-file/102772/3 "2024-08-19T12:15:45Z")

</div>

Here I created a new tokenizer  
te\_tokenizer = LlamaTokenizer.from\_pretrained(‘/content/te\_tokenizer’)

Now, I want to merge it with LLama Tokenizer. So  
llama\_tokenizer = LlamaTokenizer.from\_pretrained(‘meta-llama/Llama-2-7b-chat-hf’)

I combined these two tokenizers and saved it locally and loaded it with Llama Tokenizer  
new\_te\_tokenizer = LlamaTokenizer.from\_pretrained(‘/content/extended\_tokenizer’)

Now, I saved the tokenizer using .save\_pretrinaed()  
new\_te\_tokenizer.save\_pretrained(‘telugu\_llama2\_tokenizer’)

Then pushed it to the hub. You can visit the tokenizer here:

> **[Akirami/telugu\_llama2\_tokenizer · Hugging Face](https://huggingface.co/Akirami/telugu_llama2_tokenizer)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

The thing is, when I called the .save\_pretrained(), it only produced okenizer\_config.json, tokenizer.model and special\_tokens\_map.json. But I want the tokenizer.json file, which I can find in the original LlamaTokenizer repo.

And let’s say I want to load my tokenizer with the HuggingFace Tokenizer library. That library requires a tokenizer.json file to load it.

So the question is, how do I get the tokenizer.json for my new merged model?

---

<div class="post-metadata">

**Author:** ![mahmutc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mahmutc/32/52583_2.png) [@mahmutc](https://discuss.huggingface.co/u/mahmutc)\
**Post date:** [August 19, 2024, 2:00pm UTC](https://discuss.huggingface.co/t/save-pretrained-on-tokenizer-does-not-generate-a-tokenizer-json-file/102772/4 "2024-08-19T14:00:12Z")

</div>

From [transformers/src/transformers/tokenization\_utils\_base.py at main · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py)

```auto
# Slow tokenizers used to be saved in three separated files
SPECIAL_TOKENS_MAP_FILE = "special_tokens_map.json"
ADDED_TOKENS_FILE = "added_tokens.json"
TOKENIZER_CONFIG_FILE = "tokenizer_config.json"

# Fast tokenizers (provided by HuggingFace tokenizer's library) can be saved in a single file
FULL_TOKENIZER_FILE = "tokenizer.json"
_re_tokenizer_file = re.compile(r"tokenizer\.(.*)\.json")

```

Can you please check whether your new tokenizer is fast or not?  
`tokenizer.is_fast` should return True.