# Training a tokenizer

**URL:** <https://discuss.huggingface.co/t/training-a-tokenizer/21181>\
**Category:** Beginners\
**Created:** [August 3, 2022, 4:20am UTC](https://discuss.huggingface.co/t/training-a-tokenizer/21181 "2022-08-03T04:20:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![duynguyen236](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@duynguyen236](https://discuss.huggingface.co/u/duynguyen236)\
**Post date:** [August 3, 2022, 4:20am UTC](https://discuss.huggingface.co/t/training-a-tokenizer/21181/1 "2022-08-03T04:20:01Z")

</div>

Hi everyone,  
I have a dataset in which sentences have been segmented into words. How do I use it to train a BPE or SentencePiece tokenizer?  
Thank you

---

<div class="post-metadata">

**Author:** ![mapama247](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mapama247/32/33424_2.png) [@mapama247](https://discuss.huggingface.co/u/mapama247)\
**Post date:** [August 3, 2022, 9:04am UTC](https://discuss.huggingface.co/t/training-a-tokenizer/21181/2 "2022-08-03T09:04:00Z")

</div>

Check out [this](https://github.com/huggingface/notebooks/blob/main/examples/tokenizer_training.ipynb) notebook from huggingface’s github, or the second step of [this](https://colab.research.google.com/github/huggingface/blog/blob/main/notebooks/01_how_to_train.ipynb) other notebook about how to pretrain a LM from scratch 🙂