# Trained tokenizer API as PretrainedTokenizer

**URL:** <https://discuss.huggingface.co/t/trained-tokenizer-api-as-pretrainedtokenizer/23639>\
**Category:** 🤗Tokenizers\
**Created:** [September 26, 2022, 4:35pm UTC](https://discuss.huggingface.co/t/trained-tokenizer-api-as-pretrainedtokenizer/23639 "2022-09-26T16:35:10Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![AfonsoSousa](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@AfonsoSousa](https://discuss.huggingface.co/u/AfonsoSousa)\
**Post date:** [September 26, 2022, 4:35pm UTC](https://discuss.huggingface.co/t/trained-tokenizer-api-as-pretrainedtokenizer/23639/1 "2022-09-26T16:35:10Z")

</div>

Hi. I have trained a BPE Tokenizer successfully using:

```auto
 tokenizer = Tokenizer(BPE(unk_token=unk_token))
 trainer = BpeTrainer(special_tokens=special_tokens)
 tokenizer.train(text_files, trainer)

```

But I lack things like `tokenizer.pad_token` to access the padding token or `len(tokenizer)` to get the vocabulary size. Is this the best way to train a BPE tokenizer? How can I get the same API as PretrainedTokenizer? Thanks in advance for any help you can provide.

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](https://discuss.huggingface.co/u/lianghsun)\
**Post date:** [October 25, 2022, 8:29pm UTC](https://discuss.huggingface.co/t/trained-tokenizer-api-as-pretrainedtokenizer/23639/2 "2022-10-25T20:29:16Z")

</div>

Not pretty sure what questions you met, so I just explain how to deal with padding and get vocab size here:

1. Padding: the reason we set `[PAD]` in training phase is to tell tokenizer that we might have a special token there, but we do not tell the function about each special token at moment. Then after the training you finally get a trained tokenizer instance, then you can specify `tokenizer.enable_padding(pad_id=tokenizer.tokens_to_id('[PAD]'))` to tell the tokenizer that we want to set `[PAD]` to the roll of the padding token which we have already setup in vocab list before. After setting up `.enable_padding()`, the tokenized sentense might contain padding.
2. Vocab size: `tokenizer.get_vocab_size()`