# Implementing custom tokenizer components (normalizers, processors)

**URL:** <https://discuss.huggingface.co/t/implementing-custom-tokenizer-components-normalizers-processors/12371>\
**Category:** 🤗Tokenizers\
**Created:** [November 29, 2021, 6:06pm UTC](https://discuss.huggingface.co/t/implementing-custom-tokenizer-components-normalizers-processors/12371 "2021-11-29T18:06:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![saattrupdan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saattrupdan/32/28869_2.png) [@saattrupdan](https://discuss.huggingface.co/u/saattrupdan)\
**Post date:** [November 29, 2021, 6:06pm UTC](https://discuss.huggingface.co/t/implementing-custom-tokenizer-components-normalizers-processors/12371/1 "2021-11-29T18:06:24Z")

</div>

I’m wondering if there is an easy way to tweak the individual components of a tokenizer. Specifically, I’d like to implement a custom normalizer and post-processor.

Just to provide some context, I’m trying to train a Danish tokenizer. Danish has a lot of compound nouns (e.g., the Danish translation of “house owner” is “husejer”, with “hus” being “house” and “ejer” being “owner”), so a tokenizer should split these accordingly. A standard BPE or WordPiece can deal with this just fine.

The issue is that for _some_ compound nouns, we impose an “s” in between the two words. For instance, “birthday greeting” is “fødselsdagshilsen”, with “fødselsdag” being “birthday” and “hilsen” being “greeting”. This messes up the tokenizer completely, tokenizing it as [‘fødselsdag’, ‘shi’, ‘l’, ‘sen’] rather than the ideal [‘fødselsdag’, ‘s’, ‘hilsen’].

I think I can solve it by imposing a new special token, `<conn>`, and at the normaliser stage I check if the word is of the form `<word1>`s`<word2>` where `<word1>` and `<word2>` are known words, and if so, replaces the “s” by `<conn>`. At the post-processing stage, I then replace the `<conn>` instances with “s” again.

Long story short, is there a way to simply subclass the normaliser/processor classes to implement such behaviours?

---

<div class="post-metadata">

**Author:** ![saattrupdan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/saattrupdan/32/28869_2.png) [@saattrupdan](https://discuss.huggingface.co/u/saattrupdan)\
**Post date:** [November 30, 2021, 12:41pm UTC](https://discuss.huggingface.co/t/implementing-custom-tokenizer-components-normalizers-processors/12371/2 "2021-11-30T12:41:07Z")

</div>

For anyone else looking, this **can** be done, and it’s answered in this question:

> [@How to add additional custom pre-tokenization processing?](https://discuss.huggingface.co/t/how-to-add-additional-custom-pre-tokenization-processing/1637):
>
> I would like to add a few custom functions for pre-tokenization. For example, I would like to split numerical text from any non-numerical test. Eg ‘1000mg’ would become [‘1000’, ‘mg’]. I am trying to figure out the proper way to do this for the python binding; I think it may be a bit tricky since its a binding for the original rust version. I am looking at the pretokenizer function /huggingface/tokenizers/blob/2ccd16bf5c3dd97759d7bdf5229e2feeba314b4a/bindings/python/py\_src/tokenizers/pre\_to…