# Tokenized sequence lengths

**URL:** <https://discuss.huggingface.co/t/tokenized-sequence-lengths/15117>\
**Category:** 🤗Tokenizers\
**Created:** [February 24, 2022, 7:29pm UTC](https://discuss.huggingface.co/t/tokenized-sequence-lengths/15117 "2022-02-24T19:29:34Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](https://discuss.huggingface.co/u/jbmaxwell)\
**Post date:** [February 24, 2022, 7:29pm UTC](https://discuss.huggingface.co/t/tokenized-sequence-lengths/15117/1 "2022-02-24T19:29:35Z")

</div>

I have a strange situation where I’m trying to build a custom tokenizer for a custom “language” (encoded music). The language is designed to represent the data in a way that is compact and also reasonably human-readable. In most cases, the ideal tokenization would be essentially word-level, just separating by whitespace, but because I don’t necessarily know in advance all the possibilities, I want to avoid just forcing word-level tokenization.

What I’m finding is that, if I train a tokenizer using the basic “intro” approach on the Tokenizers doc page, e.g.:

```auto
tokenizer = Tokenizer(BPE())
corpus_root = './content'
paths = [str(x) for x in Path(corpus_root).glob("**/*.txt")]

tokenizer.pre_tokenizer = Whitespace()
from tokenizers.trainers import BpeTrainer

trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"], vocab_size=4000)
tokenizer.train(files=paths, trainer=trainer)

```

then I get pretty much the ideal vocab, and the encoding works as expected. However, this tokenizer seems to work differently to those in the Transformers library (e.g., it doesn’t recognize `tokenizer.vocab`, and so on), so using a more “stock” Transformers tokenizer seems like the safer/easier option going forward.

As a workaround, I tried saving the tokenizer files using:

```auto
tokenizer.model.save('./tokenizer/roberta_tokenizer')
tokenizer.save('./tokenizer/roberta_tokenizer/config.json')

```

and then loading this into a RobertaTokenizer, using the saved `vocab.json` and `merges.txt` files. Although this does function, what I’m noticing is that the lengths of the encoded tokenizations are dramatically different—up to 3x longer for the same `input` when using RobertaTokenizer over the `Tokenizer(BPE.from_file())` version, using the same files, e.g.:

```auto
count = len(input.split(' '))
output = tokenizer.encode(input)
print(f'{output.tokens}, count = {count}, encoded count = {len(output.ids)}')
tokenizer.model.save('./tokenizer/roberta_tokenizer')
tokenizer.save('./tokenizer/roberta_tokenizer/config.json')
roberta_tokenizer = RobertaTokenizer('./tokenizer/roberta_tokenizer/vocab.json', './tokenizer/roberta_tokenizer/merges.txt')
roberta_tokenizer.save_pretrained('./tokenizers/roberta-tokenizer')

test_tok = AutoTokenizer.from_pretrained('./tokenizers/roberta-tokenizer')
test_tok2 = Tokenizer(BPE.from_file('./tokenizer/roberta_tokenizer/vocab.json', './tokenizer/gpt2_tokenizer/merges.txt'))

test = test_tok.encode(input)
print("Roberta test: ", test)

test2 = test_tok2.encode(input)
print("Tokenizer(BPE()) test: ", test2.ids)

------- 

"Roberta test: [4000, 192, 357, 52, 5, 194, 661, 11, 33, 32, 9, 39, 9, 111, 51, 58, 6, 130, 53, 46, 51, 58, 6, 146, 6, 260, 58, 6, 1372, 7, 584, 7, 1876, 58, 6, 99, 101, 9, 37, 8, 59, 6, 151, 9, 77, 695, 11, 4001]"
"Tokenizer(BPE()) test: [939, 711, 196, 115, 131, 120, 295, 165, 233, 309, 99, 395, 100, 335, 568]"

```

I tried testing this with a small, natural-language dataset from the Datasets page, and I don’t see these dramatic differences—i.e., I only see differences of a few tokens, which could be explained by special/added tokens. So I’m assuming that it’s my “synthesized” language that is the difference. So is there a way to use `Tokenizer(BPE.from_file())`, but still have all the integration features of using, for example, `RobertaTokenizer` from the Transformers library?

I have been posting some of my saga on the Discord server, but I think this is probably a better place for something so detailed (and mysterious).

Any help would be greatly appreciated.

---

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](https://discuss.huggingface.co/u/jbmaxwell)\
**Post date:** [February 25, 2022, 12:07am UTC](https://discuss.huggingface.co/t/tokenized-sequence-lengths/15117/2 "2022-02-25T00:07:25Z")

</div>

Or, alternately, does anyone know why:

```auto
tokenizer = Tokenizer(BPE.from_file('./tokenizer/roberta_tokenizer/vocab.json', './tokenizer/roberta_tokenizer/merges.txt'))
print("vocab_size: ", tokenizer.model.vocab)

```

Fails with an error that `'tokenizers.models.BPE' object has no attribute 'vocab'`. According to the docs, it should have: [Input sequences — tokenizers documentation](https://huggingface.co/docs/tokenizers/python/latest/api/reference.html#module-tokenizers.models)

According to `tokenizers. __version__ ` I’m running 0.11.0. These docs are for 0.10.0—is vocab removed in 0.11.0? Or is something just borked in my install?

UPDATE: I gave 0.10.1 a try, just for kicks, but same error.

---

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](https://discuss.huggingface.co/u/jbmaxwell)\
**Post date:** [February 25, 2022, 2:05am UTC](https://discuss.huggingface.co/t/tokenized-sequence-lengths/15117/3 "2022-02-25T02:05:49Z")

</div>

Digging in further, it looks like the difference must be between BPE and ByteLevelBPETokenizer (i.e., RoBERTa’s tokenizer). With the former, I get the 4000 item vocab I want, but the latter only gives me a 1300 item vocab (despite indicating 4000 in the `vocab_size`).

So to get what I’m after, I have to either;

1. figure out how to get the BPE version into a tokenizer that plays nice with transformers OR
2. figure out how to get the ByteLevelBPETokenizer to learn a 4000 item vocab

---

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](https://discuss.huggingface.co/u/jbmaxwell)\
**Post date:** [February 25, 2022, 10:56pm UTC](https://discuss.huggingface.co/t/tokenized-sequence-lengths/15117/4 "2022-02-25T22:56:32Z")

</div>

Okay, I’ve made some progress with this approach:

```auto
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
from transformers import PreTrainedTokenizerFast

train = True

if train:
    tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
    trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])

    tokenizer.pre_tokenizer = Whitespace()
    tokenizer.train(files, trainer)

    tokenizer.save('./tokenizer/bpe_tokenizer/tokenizer.json')

fast_tokenizer = PreTrainedTokenizerFast(tokenizer_file="./tokenizer/bpe_tokenizer/tokenizer.json")

test = fast_tokenizer.encode(input)
print("test tokenization: ", test)

decode_test = fast_tokenizer.decode(test)
print("test decode: ", decode_test)

```

Then I modified `run_clm.py` to use `tokenizer = PreTrainedTokenizerFast(tokenizer_file=model_args.model_name_or_path)` instead of the original AutoTokenizer version.

This now works as expected. Worth noting is that you **can’t** use `PreTrainedTokenizer` (i.e., the slow version) or you’ll hit a `NotImplemented` error when trying to call `tokenizer.encode(input)`.

---

<div class="post-metadata">

**Author:** ![mmalandro](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@mmalandro](https://discuss.huggingface.co/u/mmalandro)\
**Post date:** [March 1, 2022, 2:01am UTC](https://discuss.huggingface.co/t/tokenized-sequence-lengths/15117/5 "2022-03-01T02:01:53Z")

</div>

That’s right. The methods of the tokenizers from the tokenizers library are different from the methods of the tokenizers from the transformers library. You want to use PreTrainedTokenizerFast from the transformers library to access the functionality of the transformers tokenizers.

I suspect you want to be working with a character-level BPE rather than a byte-level BPE tokenizer.

I’m also working on encoded music. We should talk 😄

---

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](https://discuss.huggingface.co/u/jbmaxwell)\
**Post date:** [March 1, 2022, 4:10pm UTC](https://discuss.huggingface.co/t/tokenized-sequence-lengths/15117/6 "2022-03-01T16:10:22Z")

</div>

Yes, that’s what I finally realized, in a kind of roundabout way.

And yes again, we should talk!

---

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](https://discuss.huggingface.co/u/jbmaxwell)\
**Post date:** [March 10, 2022, 7:12pm UTC](https://discuss.huggingface.co/t/tokenized-sequence-lengths/15117/7 "2022-03-10T19:12:02Z")

</div>

btw, Huggingface people, I’m still wondering if there’s any way to force a larger vocabulary during training? Presumably this would just be more “merging”, no? Shouldn’t there be a parameter to force a larger vocab if you want it?

EDIT: I notice I was apparently getting the 4000 word vocab when I posted this, but that’s not the case now… I request `vocab_size=4000` and I get 2026. Hmm…