# Implementing custom tokenizer components (normalizers, processors)

**URL:** <https://discuss.huggingface.co/t/implementing-custom-tokenizer-components-normalizers-processors/12371>\
**Category:** 🤗Tokenizers\
**Created:** [November 29, 2021, 6:06pm UTC](https://discuss.huggingface.co/t/implementing-custom-tokenizer-components-normalizers-processors/12371 "2021-11-29T18:06:24Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![saattrupdan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saattrupdan/32/28869_2.png) [@saattrupdan](https://discuss.huggingface.co/u/saattrupdan)\
**Post date:** [November 30, 2021, 12:41pm UTC](https://discuss.huggingface.co/t/implementing-custom-tokenizer-components-normalizers-processors/12371/2 "2021-11-30T12:41:07Z")

</div>

For anyone else looking, this **can** be done, and it’s answered in this question:

> [@How to add additional custom pre-tokenization processing?](https://discuss.huggingface.co/t/how-to-add-additional-custom-pre-tokenization-processing/1637):
>
> I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test. Eg ‘1000mg’ would become [‘1000’, ‘mg’]. I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version. I am looking at the pretokenizer function /huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py\_src/tokenizers/pre\_to…

---

_[View the full topic](https://discuss.huggingface.co/t/implementing-custom-tokenizer-components-normalizers-processors/12371)._