# Tokenizer from tokenizers library cannot be used in transformers.Trainer

**URL:** <https://discuss.huggingface.co/t/tokenizer-from-tokenizers-library-cannot-be-used-in-transformers-trainer/8773>\
**Category:** 🤗Transformers\
**Created:** [July 28, 2021, 5:04pm UTC](https://discuss.huggingface.co/t/tokenizer-from-tokenizers-library-cannot-be-used-in-transformers-trainer/8773 "2021-07-28T17:04:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aktsvigun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aktsvigun/32/19418_2.png) [@Aktsvigun](https://discuss.huggingface.co/u/Aktsvigun)\
**Post date:** [July 28, 2021, 5:04pm UTC](https://discuss.huggingface.co/t/tokenizer-from-tokenizers-library-cannot-be-used-in-transformers-trainer/8773/1 "2021-07-28T17:04:23Z")

</div>

Hi,

I am trying to train my own model with `Trainer` with a pre-trained `SentencePieceBPETokenizer` from **tokenizers** library. However, it is missing several attributes as well as methods (e.g., `pad` ), which makes it incompatible with `transformers.Trainer` . Is there an easy way to convert it to `PretrainedTokenizer` from `transformers` ?  
Thanks!

---

<div class="post-metadata">

**Author:** ![prikmm](https://avatars.discourse-cdn.com/v4/letter/p/a698b9/32.png) [@prikmm](https://discuss.huggingface.co/u/prikmm)\
**Post date:** [July 28, 2021, 5:59pm UTC](https://discuss.huggingface.co/t/tokenizer-from-tokenizers-library-cannot-be-used-in-transformers-trainer/8773/2 "2021-07-28T17:59:47Z")

</div>

If you want SentencePiece Tokenizer, you should use the [sentencepiece library](https://github.com/google/sentencepiece), then pass in the trained model as a parameter into the desired tokenizer model like T5, Bart etc. By doing this the vocab will be yours and the desired tokenizer will handle the padding, I am not sure about whether it will handle the special tokens though.

---

<div class="post-metadata">

**Author:** ![Aktsvigun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aktsvigun/32/19418_2.png) [@Aktsvigun](https://discuss.huggingface.co/u/Aktsvigun)\
**Post date:** [July 30, 2021, 11:21am UTC](https://discuss.huggingface.co/t/tokenizer-from-tokenizers-library-cannot-be-used-in-transformers-trainer/8773/3 "2021-07-30T11:21:27Z")

</div>

**sgugger** helped with the solution, we simply need  
`transformers.PretrainedTokenizerFast(tokenizer_object=my_tokenizer)`.