# Truncate the seq. not working

**URL:** <https://discuss.huggingface.co/t/truncate-the-seq-not-working/21722>\
**Category:** 🤗Transformers\
**Created:** [August 17, 2022, 7:04am UTC](https://discuss.huggingface.co/t/truncate-the-seq-not-working/21722 "2022-08-17T07:04:29Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](https://discuss.huggingface.co/u/lianghsun)\
**Post date:** [August 17, 2022, 7:04am UTC](https://discuss.huggingface.co/t/truncate-the-seq-not-working/21722/1 "2022-08-17T07:04:29Z")

</div>

Hi all, my tokenizer configurations is as follow:

```auto
tokenizer = Tokenizer(BPE(
    unk_token='[UNK]'
))
tokenizer.enable_padding(
    direction='right',
    pad_id=0,
    pad_token='[PAD]'
)
tokenizer.enable_truncation(
    max_length=512, 
    direction='right'
)
tokenizer.add_special_tokens(
    list({'bos_token': '[BOS]',
         'eos_token': '[EOS]',
         'unk_token': '[UNK]',
         'sep_token': '[SEP]',
         'pad_token': '[PAD]',
         'cls_token': '[CLS]',
         'mask_token': '[MASK]'})
)
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(
    add_prefix_space=False, 
    use_regex=False 
) 
tokenizer.post_processor = ByteLevel(trim_offsets=True)
tokenizer.decoder = decoders.ByteLevel()

trainer = BpeTrainer(
    special_tokens=['[PAD]', '[UNK]', '[EOS]', '[BOS]', '[MASK]'], 
    show_progress=True, 
    initial_alphabet=['[BOS]']
)

tokenizer.train(files=[datasets_path], trainer=trainer)

```

After testing this trained tokenizer, it works fine:

```auto
encoded_smi = tokenizer.encode('[BOS]c1ccccc1')
print(f"Encoded Tokens: {encoded_smi.ids}")
>> Encoded Tokens: [3, 49, 12, 74, 12]

decoded_smi = tokenizer.decode(encoded_smi.ids)
print(f"Decoded SMILES: {decoded_smi}")
>> Decoded SMILES: c1ccccc1

```

However, when I try to use Transformers Tokenizer API, everything is change:

```auto
from transformers import AutoTokenizer
new_tokenizer = AutoTokenizer.from_pretrained('./path/to/tokenization/')
new_tokenizer
>> PreTrainedTokenizerFast(name_or_path='./path/to/tokenization/', vocab_size=11449, model_max_len=1000000000000000019884624838656, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '[BOS]', 'eos_token': '[EOS]', 'unk_token': '[UNK]', 'sep_token': '[SEP]', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'mask_token': '[MASK]'})

```

I found out that `model_max_len` is different from the setting, however it should be `512` instead. Then I make a test on new\_tokenizer:

```auto
test_smi = 'C'*1024*128*2*2
tokens = new_tokenizer.encode(test_smi)
display(len(tokens), tokens[1], tokens[-1])
>> 
2048
6014
6014

```

It’s obvious that tokenizer didn’t truncate the seq. However if i change the code to:

```auto
len(new_tokenizer(test_smi, truncation=True, max_length=512)['input_ids'])
>> 512

```

Then everything worked fine! I wonder why `AutoTokenizer.from_pretrained()` API doesn’t keep the maximum length setting, and would it affect the Transformer training phase…?