# Pre\_tokenization

**URL:** <https://discuss.huggingface.co/t/pre-tokenization/36391>\
**Category:** 🤗Transformers\
**Created:** [April 13, 2023, 12:02am UTC](https://discuss.huggingface.co/t/pre-tokenization/36391 "2023-04-13T00:02:05Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jonatm4](https://avatars.discourse-cdn.com/v4/letter/j/2acd7d/32.png) [@jonatm4](https://discuss.huggingface.co/u/jonatm4)\
**Post date:** [April 13, 2023, 12:02am UTC](https://discuss.huggingface.co/t/pre-tokenization/36391/1 "2023-04-13T00:02:05Z")

</div>

I’m trying to pretokenize input sentences using GPT2TokenizerFast, then feed the pretokenized input through the sub-word tokenizer.

I found out that the Fast Tokenizers in the huggingface library allow access to pretokenization through:  
`tokenizer.backend_tokenizer.pre_tokenizer.pre_tokenize_str(string)`

This works to get the pretokenized input, and I thought that I could just feed this into the tokenizer with ‘is split into words’ set to ‘True’ as follows:  
`tokenizer(pretokenized_input, return_tensors='pt', is_split_into_words = True).tokens()`  
but I get an unexpected result. After pretokenization of the sentence ‘The person went to the store’ I get:  
[‘ĠThe’, ‘Ġperson’, ‘Ġwent’, ‘Ġto’, ‘Ġthe’, ‘Ġstore’]

But when I feed this through the tokenizer I get:  
[‘ĠÄ’, ‘ł’, ‘The’, ‘ĠÄ’, ‘ł’, ‘person’, ‘ĠÄ’, ‘ł’, ‘went’, ‘ĠÄ’, ‘ł’, ‘to’, ‘ĠÄ’, ‘ł’, ‘the’, ‘ĠÄ’, ‘ł’, ‘store’]

Why does it add the tokens: ‘ĠÄ’ and ‘ł’?