# Preprocessing raw text

**URL:** <https://discuss.huggingface.co/t/preprocessing-raw-text/24559>\
**Category:** 🤗Tokenizers\
**Created:** [October 17, 2022, 1:51pm UTC](https://discuss.huggingface.co/t/preprocessing-raw-text/24559 "2022-10-17T13:51:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![antoine2323231](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@antoine2323231](https://discuss.huggingface.co/u/antoine2323231)\
**Post date:** [October 17, 2022, 1:51pm UTC](https://discuss.huggingface.co/t/preprocessing-raw-text/24559/1 "2022-10-17T13:51:36Z")

</div>

When fitting a custom tokenizer, I am preprocessing raw text. I remove punctuation, but then there is no indication of the end/start of a sentence, therefore if I keep the “.” as an example that would be understood by the tokenizer model? Do you have any suggestions?

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](https://discuss.huggingface.co/u/lianghsun)\
**Post date:** [October 25, 2022, 7:42pm UTC](https://discuss.huggingface.co/t/preprocessing-raw-text/24559/2 "2022-10-25T19:42:40Z")

</div>

Hey bro, have you tried [TemplateProcessing](https://huggingface.co/docs/tokenizers/api/post-processors#tokenizers.processors.TemplateProcessing) ? You should add the following code:

```auto
tokenizer.post_processor = TemplateProcessing(
    single="[BOS] $0 [EOS]",
    special_tokens=[
        ("[BOS]", tokenizer.token_to_id("[BOS]")),
        ("[EOS]", tokenizer.token_to_id("[EOS]"))
    ]

```

Please note that you may need to add `[PAD]` through `tokenizer.enable_padding()`.

---

<div class="post-metadata">

**Author:** ![antoine2323231](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@antoine2323231](https://discuss.huggingface.co/u/antoine2323231)\
**Post date:** [October 26, 2022, 3:21pm UTC](https://discuss.huggingface.co/t/preprocessing-raw-text/24559/3 "2022-10-26T15:21:31Z")

</div>

Hey bro,  
Thanks a lot! Gonna try this out.