# What does the parameter 'clean\_up\_tokenization\_spaces' do in the tokenizer.decode function?

**URL:** <https://discuss.huggingface.co/t/what-does-the-parameter-clean-up-tokenization-spaces-do-in-the-tokenizer-decode-function/17399>\
**Category:** Beginners\
**Created:** [May 2, 2022, 2:28am UTC](https://discuss.huggingface.co/t/what-does-the-parameter-clean-up-tokenization-spaces-do-in-the-tokenizer-decode-function/17399 "2022-05-02T02:28:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ivnle](https://avatars.discourse-cdn.com/v4/letter/i/b5a626/32.png) [@ivnle](https://discuss.huggingface.co/u/ivnle)\
**Post date:** [May 2, 2022, 2:28am UTC](https://discuss.huggingface.co/t/what-does-the-parameter-clean-up-tokenization-spaces-do-in-the-tokenizer-decode-function/17399/1 "2022-05-02T02:28:28Z")

</div>

As titled. The [documentation](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.batch_decode.clean_up_tokenization_spaces) is a bit vague.

> Whether or not to clean up the tokenization spaces.

---

<div class="post-metadata">

**Author:** ![morenolq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/morenolq/32/55542_2.png) [@morenolq](https://discuss.huggingface.co/u/morenolq)\
**Post date:** [December 5, 2022, 6:03pm UTC](https://discuss.huggingface.co/t/what-does-the-parameter-clean-up-tokenization-spaces-do-in-the-tokenizer-decode-function/17399/2 "2022-12-05T18:03:34Z")

</div>

It should remove space artifacts inserted while encoding the sequence. E.g., if you have `state-of-the-art` it will be encoded as `state - of - the - art`. The cleanup should remove those spaces between `-`. Hope it helps!

---

<div class="post-metadata">

**Author:** ![gspeter0](https://avatars.discourse-cdn.com/v4/letter/g/7ea924/32.png) [@gspeter0](https://discuss.huggingface.co/u/gspeter0)\
**Post date:** [July 8, 2025, 9:45am UTC](https://discuss.huggingface.co/t/what-does-the-parameter-clean-up-tokenization-spaces-do-in-the-tokenizer-decode-function/17399/3 "2025-07-08T09:45:04Z")

</div>

But may be ‘-’ these like tokens is important for response or output

You sure about that