# How to get sp\_model variable from T5Tokenizer?

**URL:** <https://discuss.huggingface.co/t/how-to-get-sp-model-variable-from-t5tokenizer/25171>\
**Category:** 🤗Tokenizers\
**Created:** [October 29, 2022, 7:06am UTC](https://discuss.huggingface.co/t/how-to-get-sp-model-variable-from-t5tokenizer/25171 "2022-10-29T07:06:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![uunal](https://avatars.discourse-cdn.com/v4/letter/u/a88e4f/32.png) [@uunal](https://discuss.huggingface.co/u/uunal)\
**Post date:** [October 29, 2022, 7:06am UTC](https://discuss.huggingface.co/t/how-to-get-sp-model-variable-from-t5tokenizer/25171/1 "2022-10-29T07:06:34Z")

</div>

Is there a way to acquire sp\_model variable from T5Tokenizer? Related code can be found at `https://github.com/huggingface/transformers/blob/2e35bac4e73558d334ea5bbf96a1116f7c0d7fb3/src/transformers/models/t5/tokenization_t5.py#L155`  
When we initiate tokenizer with ex. “t5-base”, from\_pretrained function only returns fast version which does not have sp\_model variable.

My aim is to initiate text.SentencepieceTokenizer from sp\_model. If there is a way, this will help converting t5 tokenizer easily into tensorflow.

Or if there is a alternative way, I’ll be glad to learn 🙂

Thanks all in advance,

---

<div class="post-metadata">

**Author:** ![uunal](https://avatars.discourse-cdn.com/v4/letter/u/a88e4f/32.png) [@uunal](https://discuss.huggingface.co/u/uunal)\
**Post date:** [October 29, 2022, 6:53pm UTC](https://discuss.huggingface.co/t/how-to-get-sp-model-variable-from-t5tokenizer/25171/2 "2022-10-29T18:53:13Z")

</div>

I solved it this way,

```auto
from transformers import AutoTokenizer
import sentencepiece as spm
from tensorflow_text import SentencepieceTokenizer

tokenizer = AutoTokenizer.from_pretrained("t5-base")

sp_model = spm.SentencePieceProcessor()
sp_model.Load(tokenizer.vocab_file)

sp_proto = sp_model.serialized_model_proto()

tf_sp = SentencepieceTokenizer(
    model=sp_proto,
    alpha=0.1,
    nbest_size=0,
    add_bos=False,
    add_eos=True,
    reverse=False
)
#now you can tokenize or detokenize with 
input_ids = tf_sp.tokenize("some text")
"some text" as tensor = tf_sp.detokenize(input_ids)

```