# Error loading tokenizer from local checkpoint directory

**URL:** <https://discuss.huggingface.co/t/error-loading-tokenizer-from-local-checkpoint-directory/44428>\
**Category:** 🤗Tokenizers\
**Created:** [June 25, 2023, 12:43am UTC](https://discuss.huggingface.co/t/error-loading-tokenizer-from-local-checkpoint-directory/44428 "2023-06-25T00:43:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmatekenya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dmatekenya/32/16995_2.png) [@dmatekenya](https://discuss.huggingface.co/u/dmatekenya)\
**Post date:** [June 25, 2023, 12:43am UTC](https://discuss.huggingface.co/t/error-loading-tokenizer-from-local-checkpoint-directory/44428/1 "2023-06-25T00:43:47Z")

</div>

I’m fine-tuning Whisper for a low-resource language (Chichewa) and following [this tutorial](https://huggingface.co/blog/fine-tune-whisper). One change I have made is to provide a local directory to save the model instead of pushing to Hub. When its time to use the fine-tuned model using the pipeline module, I’m getting this error:  
`Can't load tokenizer for '/content/drive/My Drive/Chichewa-ASR/models/whisper-small-chich/checkpoint-1000. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/content/drive/My Drive/Chichewa-ASR/models/whisper-small-chich/checkpoint-1000' is the correct path to a directory containing all relevant files for a WhisperTokenizer tokenizer.`  
When I check the model repository used by the tutorial author, I see that it has several files I don’t have in my model checkpoint directory such as `tokenizer_config.json`. I’m just wondering how do I ensure these files are saved in the local model checkpoint directories?

---

<div class="post-metadata">

**Author:** ![www007](https://avatars.discourse-cdn.com/v4/letter/w/3da27b/32.png) [@www007](https://discuss.huggingface.co/u/www007)\
**Post date:** [November 1, 2023, 2:45am UTC](https://discuss.huggingface.co/t/error-loading-tokenizer-from-local-checkpoint-directory/44428/2 "2023-11-01T02:45:20Z")

</div>

I had the same problem, have you solved it?

---

<div class="post-metadata">

**Author:** ![Unspoiled-Egg](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/unspoiled-egg/32/25070_2.png) [@Unspoiled-Egg](https://discuss.huggingface.co/u/Unspoiled-Egg)\
**Post date:** [April 4, 2024, 7:12am UTC](https://discuss.huggingface.co/t/error-loading-tokenizer-from-local-checkpoint-directory/44428/3 "2024-04-04T07:12:28Z")

</div>

@sanchit-gandhi can you help with this, I have similar problem

---

<div class="post-metadata">

**Author:** ![Mutturaj](https://avatars.discourse-cdn.com/v4/letter/m/4af34b/32.png) [@Mutturaj](https://discuss.huggingface.co/u/Mutturaj)\
**Post date:** [May 13, 2024, 1:47am UTC](https://discuss.huggingface.co/t/error-loading-tokenizer-from-local-checkpoint-directory/44428/4 "2024-05-13T01:47:49Z")

</div>

I had same problem,  
In tried to copy the tokenizer\_config.json file from whisper model output directory to checkpoint directory

Then i load the model from checkpoint directory

It worked form me