# Padding not transferring when loading a tokenizer trained via the tokenizers library into transformers **URL:** \ **Category:** 🤗Tokenizers\ **Created:** [June 12, 2023, 12:27am UTC](https://discuss.huggingface.co/t/padding-not-transferring-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42872 "2023-06-12T00:27:46Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![abhinavp](https://avatars.discourse-cdn.com/v4/letter/a/e19adc/32.png) [@abhinavp](https://discuss.huggingface.co/u/abhinavp)\ **Post date:** [June 12, 2023, 12:27am UTC](https://discuss.huggingface.co/t/padding-not-transferring-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42872/1 "2023-06-12T00:27:46Z")
Reposting this here from the [transformers forum](https://discuss.huggingface.co/t/padding-not-working-when-loading-a-tokenizer-trained-via-the-tokenizers-library-into-transformers/42326/1) because I got no answer there: Hi, I trained a simple WhitespaceSplit/WordLevel tokenizer using the `tokenizers` library. I added padding by calling `enable_padding(pad_token="")` on the Tokenizer instance. Then I saved it to a JSON file and then loaded it into transformers using [the instructions here](https://huggingface.co/docs/transformers/fast_tokenizers): ```auto fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json") ``` When using the `tokenizers.Tokenizer` object directly, `encode` correctly adds the padding tokens. However, if I try padding when tokenizing using the `PreTrainedTokenizerFast` instance, I get the exception: ```auto ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`. ``` Sure enough, if I follow the instructions and add the pad token as a special token, it works. But I want the tokenizer to work _out of the box_, exactly as the equivalent `tokenizer.Tokenizer` instance does, including in terms of padding behavior. Why is this not the case? Why do I have to enable padding for the `tokenizer.Tokenizer` instance, and then _again_ for the `PreTrainedTokenizerFast` instance? Am I doing something wrong or missing something? To reproduce the problem, you can use the code below. Most of it is from the [tokenizers Quicktour](https://huggingface.co/docs/tokenizers/quicktour), so you’ll need to download the data files as per the instructions there (or modify `files` if using your own files). The rest is from the official transformers docs on [how to load a tokenizer from `tokenizers` into `transformers`](https://huggingface.co/docs/transformers/fast_tokenizers): ```py from tokenizers import BpeTrainer, Tokenizer from tokenizers.models import BPE from tokenizers.pre_tokenizers import Whitespace from transformers import PreTrainedTokenizerFast files = [f"data/wikitext-103-raw/wiki.{split}.raw" for split in ["test", "train", "valid"]] sentences = ["Hello, y'all!", "How are you 😁 ?"] tokenizer = Tokenizer(BPE(unk_token="[UNK]")) trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"]) tokenizer.pre_tokenizer = Whitespace() tokenizer.train(files, trainer) # Enable padding tokenizer.enable_padding(pad_id=3, pad_token="[PAD]") # Now use this tokenizer to tokenize a couple of sentences. output = tokenizer.encode_batch(sentences) # The output is padded, as it should be: print(output[0].tokens) # ['Hello', ',', 'y', "'", 'all', '!'] print(output[1].tokens) # ['How', 'are', 'you', '[UNK]', '?', '[PAD]'] # But now let's say we load the tokenizer into transformers- let's try loading it directly from the tokenizer object: fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer) # Tokenize two strings of different token length with padding fast_output = fast_tokenizer(sentences, padding=True) ``` This gives us the error: ```auto Using pad_token, but it is not set yet. Traceback (most recent call last): File "", line 1, in File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2548, in __call__ encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs) File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2634, in _call_one return self.batch_encode_plus( File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2816, in batch_encode_plus padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies( File "/Users/apatil/anaconda3/envs/lm-training/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 2453, in _get_padding_truncation_strategies raise ValueError( ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`. ``` We can resolve the issue by explicitly specifying the special tokens when initializing the `PreTrainedTokenizerFast`: ```py fast_tokenizer = PreTrainedTokenizerFast(tokenizer_object=tokenizer, pad_token="[PAD]", unk_token="[UNK]") # Now padding works as expected fast_output = fast_tokenizer(sentences, padding=True) print(fast_output[0].tokens) # ['Hello', ',', 'y', "'", 'all', '!'] print(fast_output[1].tokens) # ['How', 'are', 'you', '[UNK]', '?', '[PAD]'] ``` The code above uses the `tokenizer_object` parameter to load the fast tokenizer as a `PreTrainedTokenizerFast` instance, but as you can confirm for yourselves, the same behavior occurs if you first save the tokenizer to file, then load it into `PreTrainedTokenizerFast` using the `tokenizer_file` parameter instead. **Bottom line:** I don’t understand, if the padding information is already in the tokenizer (or in the saved tokenizer config file), why I should need to explicitly specify the padding token again when transferring the tokenizer. This introduces a lot of totally unnecessary friction into what should be a painless process. The tokenizer object/config should be self-contained. I should not have to re-hardcode what the padding token is in the code that loads it into `transformers`, if that information is already encapsulated in the tokenizer object or its saved config file, any more than I should need to specify the vocab file, or the pretokenizers to use, etc. That’s the whole point of the tokenizer object/config file: to uniquely determine the behavior the tokenizer. Am I doing something wrong, or is this just how this works?