# T5 model tokenizer

**URL:** <https://discuss.huggingface.co/t/t5-model-tokenizer/23640>\
**Category:** 🤗Tokenizers\
**Created:** [September 26, 2022, 4:38pm UTC](https://discuss.huggingface.co/t/t5-model-tokenizer/23640 "2022-09-26T16:38:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![antoine2323231](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@antoine2323231](https://discuss.huggingface.co/u/antoine2323231)\
**Post date:** [September 26, 2022, 4:38pm UTC](https://discuss.huggingface.co/t/t5-model-tokenizer/23640/1 "2022-09-26T16:38:26Z")

</div>

T5 models are using BPE tokenizers? Is it possible to use another type of tokenizer along a T5 model, or not because they are designed to work with BPE?

---

<div class="post-metadata">

**Author:** ![michaelwechner](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michaelwechner/32/5514_2.png) [@michaelwechner](https://discuss.huggingface.co/u/michaelwechner)\
**Post date:** [September 27, 2022, 1:39pm UTC](https://discuss.huggingface.co/t/t5-model-tokenizer/23640/2 "2022-09-27T13:39:40Z")

</div>

AFAIK T5 is using SentencePiece [T5](https://huggingface.co/docs/transformers/model_doc/t5#transformers.T5Tokenizer) which has BPE implemented [GitHub - google/sentencepiece: Unsupervised text tokenizer for Neural Network-based text generation.](https://github.com/google/sentencepiece) and therefore depends on this.

Why would you like to use another tokenizer?

---

<div class="post-metadata">

**Author:** ![michaelwechner](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/michaelwechner/32/5514_2.png) [@michaelwechner](https://discuss.huggingface.co/u/michaelwechner)\
**Post date:** [September 29, 2022, 11:54am UTC](https://discuss.huggingface.co/t/t5-model-tokenizer/23640/3 "2022-09-29T11:54:09Z")

</div>

The question got answered on [Discord](https://discord.com/channels/879548962464493619/1023996386968092783/1024626562743676958)

If you’re training from scratch, then you would typically train a tokenizer on your own data, in which case you can choose which tokenizer training algorithm (BPE, WordPiece or UnigramLM if you’re using 🤗 tokenizers) and how to preprocess the data before tokenizing it. I can recommend this chapter of the HF course to learn more about tokenizers: [Introduction - Hugging Face Course](https://huggingface.co/course/chapter6/1)