# Issue with tokenizer.tokenize

**URL:** <https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891>\
**Category:** 🤗Tokenizers\
**Created:** [November 4, 2020, 1:37pm UTC](https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891 "2020-11-04T13:37:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![xiakun](https://avatars.discourse-cdn.com/v4/letter/x/f19dbf/32.png) [@xiakun](https://discuss.huggingface.co/u/xiakun)\
**Post date:** [November 4, 2020, 1:37pm UTC](https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891/1 "2020-11-04T13:37:54Z")

</div>

when I test tokenizer.tokenize(‘how do you do’) （RobertaTokenizer in pytorch\_transformers.tokenization\_roberta.py ）, it returns [‘how’, ‘Ġ’, ‘do’, ‘Ġ’, ‘you’, ‘Ġ’, ‘do’], wants to know where is the wrong

---

<div class="post-metadata">

**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\
**Post date:** [November 5, 2020, 8:19am UTC](https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891/2 "2020-11-05T08:19:12Z")

</div>

There is some discussion in this [therad](https://stackoverflow.com/questions/61134275/difficulty-in-understanding-the-tokenizer-used-in-roberta-model) and [this](https://stackoverflow.com/questions/62422590/do-i-need-to-pre-tokenize-the-text-first-before-using-huggingfaces-robertatoken). Perhaps it helps?

---

<div class="post-metadata">

**Author:** ![facehugger2020](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/facehugger2020/32/3052_2.png) [@facehugger2020](https://discuss.huggingface.co/u/facehugger2020)\
**Post date:** [November 16, 2020, 7:21pm UTC](https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891/3 "2020-11-16T19:21:35Z")

</div>

See also [this post](https://discuss.huggingface.co/t/tokenizer-decoding-using-bert-roberta-xlnet-gpt2/1128) in the forum.

---

<div class="post-metadata">

**Author:** ![facehugger2020](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/facehugger2020/32/3052_2.png) [@facehugger2020](https://discuss.huggingface.co/u/facehugger2020)\
**Post date:** [November 16, 2020, 7:23pm UTC](https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891/4 "2020-11-16T19:23:36Z")

</div>

> where is the wrong

That’s a new one. I haven’t seen that expression before. LOL