# Can't set pad\_token by adding special token to Llama's tokenizer **URL:** \ **Category:** 🤗Transformers\ **Created:** [November 2, 2023, 7:26am UTC](https://discuss.huggingface.co/t/cant-set-pad-token-by-adding-special-token-to-llamas-tokenizer/60785 "2023-11-02T07:26:03Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![phosseini](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/phosseini/32/26690_2.png) [@phosseini](https://discuss.huggingface.co/u/phosseini)\ **Post date:** [November 2, 2023, 7:26am UTC](https://discuss.huggingface.co/t/cant-set-pad-token-by-adding-special-token-to-llamas-tokenizer/60785/1 "2023-11-02T07:26:03Z")
I’m trying to fine-tune a Llama model. As we know in the model’s tokenizer by default the `pad_token` is not set which will result in getting an error when we want to pad input samples. As suggested by others, I’m trying to add the `pad_token` by adding a special token like the following: ```auto tokenizer = AutoTokenizer.from_pretrained(args.model_path) tokenizer.add_special_tokens({'pad_token': '[PAD]'}) # Avoiding mismatch between model input and tokenizer length size model.resize_token_embeddings(len(tokenizer)) ``` And the model in this case is `meta-llama/Llama-2-7b-hf`. When I check the tokenizer after running my code, I don’t see `[PAD]` added to the tokenizer, and in the `special_tokens_map.json` I see that `"pad_token": ""`. I wonder why this happens. Also, once I fine-tune my model (that part does run successfully,) I wonder when I’m loading my model, do I need to add the padding token again or will the loader automatically load the updated tokenizer with added padding token? ---
**Author:** ![vikram0711](https://avatars.discourse-cdn.com/v4/letter/v/8491ac/32.png) [@vikram0711](https://discuss.huggingface.co/u/vikram0711)\ **Post date:** [November 6, 2023, 5:51am UTC](https://discuss.huggingface.co/t/cant-set-pad-token-by-adding-special-token-to-llamas-tokenizer/60785/2 "2023-11-06T05:51:55Z")
I have the same exact questions as you & evidently there isn’t an answer to this anywhere on this forum. But I do see that my pad token has been updated after executing: `print(repr(tokenizer.pad_token))` ---
**Author:** ![phosseini](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/phosseini/32/26690_2.png) [@phosseini](https://discuss.huggingface.co/u/phosseini)\ **Post date:** [November 6, 2023, 6:25pm UTC](https://discuss.huggingface.co/t/cant-set-pad-token-by-adding-special-token-to-llamas-tokenizer/60785/3 "2023-11-06T18:25:04Z")
When I load the tokenizer after fine-tuning my model, the pad token is set, and `tokenizer.pad_token` shows `` (even though I expect it to be `[PAD]`). Now the problem is that when I want to do inference, I get the following error: `ValueError: Cannot handle batch sizes > 1 if no padding token is defined.` but the pad token is clearly set (regardless of what value). I don’t know why this happens! ---
**Author:** ![vikram0711](https://avatars.discourse-cdn.com/v4/letter/v/8491ac/32.png) [@vikram0711](https://discuss.huggingface.co/u/vikram0711)\ **Post date:** [November 8, 2023, 5:15am UTC](https://discuss.huggingface.co/t/cant-set-pad-token-by-adding-special-token-to-llamas-tokenizer/60785/4 "2023-11-08T05:15:21Z")
After fine tuning, along with the fine tuned model, are you also saving the updated tokenizer (with [PAD]) ? I think you’re supposed to save both and during inference load both and then that probably will not cause any issue. Unless of course, you’re already doing this and the issue still persists. ---
**Author:** ![IqB](https://avatars.discourse-cdn.com/v4/letter/i/ccd318/32.png) [@IqB](https://discuss.huggingface.co/u/IqB)\ **Post date:** [August 12, 2024, 6:17pm UTC](https://discuss.huggingface.co/t/cant-set-pad-token-by-adding-special-token-to-llamas-tokenizer/60785/5 "2024-08-12T18:17:57Z")
Hi, it will be better, if you show steps me how to write in code. I am little confused