# Tokenizer post\_processor help

**URL:** <https://discuss.huggingface.co/t/tokenizer-post-processor-help/21926>\
**Category:** 🤗Tokenizers\
**Created:** [August 22, 2022, 4:17pm UTC](https://discuss.huggingface.co/t/tokenizer-post-processor-help/21926 "2022-08-22T16:17:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![smh36](https://avatars.discourse-cdn.com/v4/letter/s/dfb087/32.png) [@smh36](https://discuss.huggingface.co/u/smh36)\
**Post date:** [August 22, 2022, 4:17pm UTC](https://discuss.huggingface.co/t/tokenizer-post-processor-help/21926/1 "2022-08-22T16:17:01Z")

</div>

I’m struggling to get the post\_processor to work for tokenisation. Can anyone point me in the right direction?

Here’s the minimal example

```auto
from transformers import AutoTokenizer
from tokenizers.processors import TemplateProcessing

tokenizer = AutoTokenizer.from_pretrained( 'bert-base-uncased' )

tokenizer.post_processor = TemplateProcessing(
    single="[CLS] [CLS] $0 [SEP] [CLS]",
    pair="[CLS] $A [SEP] $B:1 [SEP]:1",
    special_tokens=[
        ("[CLS]", tokenizer.cls_token_id), 
        ("[SEP]", tokenizer.sep_token_id)
    ],
)

text_string = "The cat sat on the mat."

tokens = tokenizer( text_string )

print( tokenizer.decode( tokens.token_ids ) )
# OUTPUT: [CLS] the cat sat on the mat. [SEP]

```

However, I would expect to see extra [CLS] flags at the beginning and the end. Where am I going wrong?

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](https://discuss.huggingface.co/u/lianghsun)\
**Post date:** [October 27, 2022, 7:48am UTC](https://discuss.huggingface.co/t/tokenizer-post-processor-help/21926/2 "2022-10-27T07:48:03Z")

</div>

Hi @smh36, I think a lot of people confuse [HF Transformers Tokenizer API](https://huggingface.co/docs/transformers/main_classes/tokenizer#tokenizer) with [HF Tokenizers](https://huggingface.co/docs/tokenizers/index) (so am I in the first time 😂 ). _HF Tokenizers_ train new vocabularies and tokenizer, and you may design customized tokenization flow with **Normalization** , **Pre-tokenization** , **Model** , **Post-tokenization** , and …etc. In contrast, _HF Transformers Tokenizer API_ loads **pre-trained** tokenizer from hub or local files. So it’s clearly if you would want to re-design the pre-trained tokenizer, you should use _HF Tokenizers_. The following code should help you 🤗

```auto
from tokenizers import Tokenizer
from tokenizers.processors import TemplateProcessing

tokenizer = Tokenizer.from_pretrained("bert-base-uncased") # use HF Tokenizers instead
tokenizer.post_processor = TemplateProcessing(
    single="[CLS] $A [CLS] [SEP]",
    special_tokens=[
        ("[CLS]", tokenizer.token_to_id("[CLS]")), 
        ("[SEP]", tokenizer.token_to_id("[SEP]")),
    ],
)

tokens = tokenizer.encode('Hi there').tokens
print(tokens) 
# > ['[CLS]', 'hi', 'there', '[CLS]', '[SEP]']

from transformers import PreTrainedTokenizerFast

_tokenizer = PreTrainedTokenizerFast(
    tokenizer_object=tokenizer,
    # Load speacil tokens manually
    # https://huggingface.co/course/chapter6/8?fw=pt
    unk_token = '[UNK]',
    sep_token = '[SEP]',
    pad_token = '[PAD]',
    cls_token = '[CLS]',
    mask_token = '[MASK]',
    model_max_length=128, # same as the block size of the model
)

print(_tokenizer('Hi there').input_ids)
# [101, 7632, 2045, 101, 102]

```