# Issues with BPE tokenizer

**URL:** <https://discuss.huggingface.co/t/issues-with-bpe-tokenizer/70381>\
**Category:** 🤗Tokenizers\
**Created:** [January 24, 2024, 8:49am UTC](https://discuss.huggingface.co/t/issues-with-bpe-tokenizer/70381 "2024-01-24T08:49:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Imran1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imran1/32/18779_2.png) [@Imran1](https://discuss.huggingface.co/u/Imran1)\
**Post date:** [January 24, 2024, 8:49am UTC](https://discuss.huggingface.co/t/issues-with-bpe-tokenizer/70381/1 "2024-01-24T08:49:57Z")

</div>

I attempted to train the BPE Hugging Face tokenizer for Pashto from scratch, but it is not decoding words correctly.

---

<div class="post-metadata">

**Author:** ![malikarumi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/malikarumi/32/31844_2.png) [@malikarumi](https://discuss.huggingface.co/u/malikarumi)\
**Post date:** [January 24, 2024, 3:56pm UTC](https://discuss.huggingface.co/t/issues-with-bpe-tokenizer/70381/2 "2024-01-24T15:56:05Z")

</div>

No one can help you with such limited information.

1. I am unfamiliar with Pashto. Does the official documentation say anything about BPE?
2. Is it possible that Pashto has already been pre trained with a different tokenizer?
3. Post your code so we can understand what the software is trying to tell you.

---

<div class="post-metadata">

**Author:** ![Imran1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imran1/32/18779_2.png) [@Imran1](https://discuss.huggingface.co/u/Imran1)\
**Post date:** [January 24, 2024, 4:09pm UTC](https://discuss.huggingface.co/t/issues-with-bpe-tokenizer/70381/3 "2024-01-24T16:09:38Z")

</div>

I solve the issues. 🤢