# Issue in loading the saved tokenizer

**URL:** <https://discuss.huggingface.co/t/issue-in-loading-the-saved-tokenizer/67978>\
**Category:** 🤗Tokenizers\
**Created:** [January 4, 2024, 1:54pm UTC](https://discuss.huggingface.co/t/issue-in-loading-the-saved-tokenizer/67978 "2024-01-04T13:54:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![hadnoor](https://avatars.discourse-cdn.com/v4/letter/h/aeb1de/32.png) [@hadnoor](https://discuss.huggingface.co/u/hadnoor)\
**Post date:** [January 4, 2024, 1:54pm UTC](https://discuss.huggingface.co/t/issue-in-loading-the-saved-tokenizer/67978/1 "2024-01-04T13:54:07Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/f/5/f5025d6c1a84495a18d2ec6d6cb94b3be711a482.png)

---

<div class="post-metadata">

**Author:** ![hadnoor](https://avatars.discourse-cdn.com/v4/letter/h/aeb1de/32.png) [@hadnoor](https://discuss.huggingface.co/u/hadnoor)\
**Post date:** [January 4, 2024, 1:54pm UTC](https://discuss.huggingface.co/t/issue-in-loading-the-saved-tokenizer/67978/2 "2024-01-04T13:54:43Z")

</div>

i was using the following code :

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from\_pretrained(“Salesforce/xgen-7b-8k-base”,trust\_remote\_code=True)

tokenizer.pad\_token = tokenizer.eos\_token

tokenizer.save\_pretrained(“toksav”)

loaded\_tokenizer = AutoTokenizer.from\_pretrained(“toksav”)