# Issue with tokenizer.tokenize

**URL:** <https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891>\
**Category:** 🤗Tokenizers\
**Created:** [November 4, 2020, 1:37pm UTC](https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891 "2020-11-04T13:37:54Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![facehugger2020](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/facehugger2020/32/3052_2.png) [@facehugger2020](https://discuss.huggingface.co/u/facehugger2020)\
**Post date:** [November 16, 2020, 7:21pm UTC](https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891/3 "2020-11-16T19:21:35Z")

</div>

See also [this post](https://discuss.huggingface.co/t/tokenizer-decoding-using-bert-roberta-xlnet-gpt2/1128) in the forum.

---

_[View the full topic](https://discuss.huggingface.co/t/issue-with-tokenizer-tokenize/1891)._