# Padding and truncation for custom tokenizer

**URL:** <https://discuss.huggingface.co/t/padding-and-truncation-for-custom-tokenizer/30180>\
**Category:** 🤗Tokenizers\
**Created:** [January 22, 2023, 5:34pm UTC](https://discuss.huggingface.co/t/padding-and-truncation-for-custom-tokenizer/30180 "2023-01-22T17:34:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![re90](https://avatars.discourse-cdn.com/v4/letter/r/b9e5f3/32.png) [@re90](https://discuss.huggingface.co/u/re90)\
**Post date:** [January 22, 2023, 5:34pm UTC](https://discuss.huggingface.co/t/padding-and-truncation-for-custom-tokenizer/30180/1 "2023-01-22T17:34:30Z")

</div>

Hi,

Is there a way to ensure that my custom tokenizer pads and tunicates my inputs?

This is my code:

```auto
# Tokenizer
tokenizer = Tokenizer(models.WordPiece())

spl_tokens = ["[UNK]", "[SEP]", "[MASK]", "[CLS]", "[PAD]"] # special tokens
trainer = trainers.WordPieceTrainer(special_tokens = spl_tokens)

tokenizer.normalizer = normalizers.BertNormalizer(lowercase=False)

pre_tokenizer = CharDelimiterSplit(" ")

tokenizer.pre_tokenizer = pre_tokenizer

tokenizer.decoder = decoders.WordPiece()

tokenizer.train([file_path], trainer) # training the tokenzier

cls_token_id = tokenizer.token_to_id("[CLS]")
sep_token_id = tokenizer.token_to_id("[SEP]")
print("[CLS] id = ", cls_token_id, ", [SEP] id = ", sep_token_id)

tokenizer.post_processor = processors.TemplateProcessing(
    single=f"[CLS]:0 $0 [SEP]:0",
    special_tokens=[
        ("[CLS]", cls_token_id),
        ("[SEP]", sep_token_id),
    ],
)

tokenizer.save("tokenizer-trained.json")

```

I can’t seem to get my model to behave like the tokenizer in this example:

> **[Preprocess](https://huggingface.co/docs/transformers/preprocessing#truncation)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![re90](https://avatars.discourse-cdn.com/v4/letter/r/b9e5f3/32.png) [@re90](https://discuss.huggingface.co/u/re90)\
**Post date:** [January 22, 2023, 6:01pm UTC](https://discuss.huggingface.co/t/padding-and-truncation-for-custom-tokenizer/30180/2 "2023-01-22T18:01:37Z")

</div>

Also, I’ve tried wrapping it inside the `BertTokenizerFast` object and calling in in the following way:

```auto
new_tokenizer = BertTokenizerFast(tokenizer_object=tokenizer, max_len=1024)

new_tokenizer.encode_plus(example_str, padding=True, truncation=True, add_special_tokens=True)

```

It still doesn’t seem to work