# Issue with post-processing

**URL:** <https://discuss.huggingface.co/t/issue-with-post-processing/2703>\
**Category:** 🤗Tokenizers\
**Created:** [December 8, 2020, 4:49pm UTC](https://discuss.huggingface.co/t/issue-with-post-processing/2703 "2020-12-08T16:49:20Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![YBS](https://avatars.discourse-cdn.com/v4/letter/y/7993a0/32.png) [@YBS](https://discuss.huggingface.co/u/YBS)\
**Post date:** [December 8, 2020, 4:49pm UTC](https://discuss.huggingface.co/t/issue-with-post-processing/2703/1 "2020-12-08T16:49:20Z")

</div>

I trained a new BertWordPieceTokenizer from scratch, using the same code from the example given in the docs. Then, I created a new TemplateProcessing object & assigned it as the tokenizer’s PostProcessor in order to add [CLS] and [SEP] tokens (also using the example code). However, when I encode sentences with the tokenizer, it doesn’t preform any post-processing.

**Code:**

> from tokenizers import BertWordPieceTokenizer  
> from tokenizers.processors import TemplateProcessing
> 
> corpus = “./corpus.txt”
> 
> tokenizer = BertWordPieceTokenizer(  
> clean\_text=True,  
> handle\_chinese\_chars=False,  
> strip\_accents=True,  
> lowercase=True,  
> )
> 
> tokenizer.train(  
> corpus,  
> vocab\_size=32000,  
> min\_frequency=2,  
> show\_progress=True,  
> special\_tokens=[“[UNK]”, “[CLS]”, “[SEP]”],  
> limit\_alphabet=1000,  
> wordpieces\_prefix=“##”,  
> )
> 
> tokenizer.post\_processor = TemplateProcessing(  
> single=“[CLS] $A [SEP]”,  
> pair=“[CLS] $A [SEP] $B:1 [SEP]:1”,  
> special\_tokens=[  
> (“[CLS]”, tokenizer.token\_to\_id(“[CLS]”)),  
> (“[SEP]”, tokenizer.token\_to\_id(“[SEP]”)),  
> ],  
> )
> 
> output = tokenizer.encode(“Hello, y’all! How are you 😁 ?”)  
> print(output.tokens)

**Output:**

> [‘hello’, ‘,’, ‘y’, “'”, ‘all’, ‘!’, ‘how’, ‘are’, ‘you’, ‘[UNK]’, ‘?’]

---

<div class="post-metadata">

**Author:** ![rbawden](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rbawden/32/28077_2.png) [@rbawden](https://discuss.huggingface.co/u/rbawden)\
**Post date:** [June 15, 2022, 8:09am UTC](https://discuss.huggingface.co/t/issue-with-post-processing/2703/2 "2022-06-15T08:09:08Z")

</div>

Hi there! Did you end up finding a solution to the problem?