# Trained a tokenizer from scratch but problem when loading

**URL:** <https://discuss.huggingface.co/t/trained-a-tokenizer-from-scratch-but-problem-when-loading/57783>\
**Category:** 🤗Transformers\
**Created:** [October 8, 2023, 8:17am UTC](https://discuss.huggingface.co/t/trained-a-tokenizer-from-scratch-but-problem-when-loading/57783 "2023-10-08T08:17:34Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![velocivirus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/velocivirus/32/20159_2.png) [@velocivirus](https://discuss.huggingface.co/u/velocivirus)\
**Post date:** [October 8, 2023, 8:17am UTC](https://discuss.huggingface.co/t/trained-a-tokenizer-from-scratch-but-problem-when-loading/57783/1 "2023-10-08T08:17:34Z")

</div>

Hi, I trained a tokenizer from scratch for raw wikitext103 using the code:

```auto
from tokenizers import Tokenizer, decoders
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace, ByteLevel
from tokenizers.processors import TemplateProcessing

tokenizer = Tokenizer(BPE(unk_token="<unk>"))
trainer = BpeTrainer(vocab_size=args.vocab_size,
                         special_tokens=['<s>', '<pad>', '</s>', '<unk>'], 
                         min_frequency=args.min_frequency,
                         show_progress=True,
                         )

tokenizer.post_processor = TemplateProcessing(
    single="<s> $A </s>",
    pair="<s> $A </s> <s> $B </s>",
    special_tokens=[("<s>", 0), ("</s>", 2)], # bos: bos id, eos: eos id
)
tokenizer.train(files, trainer)
tokenizer.save(args.output_path, pretty=True)

```

and it was saved to `tokenizer.json`. However, when I was trying to follow the tutorial to load the tokenizer by doing

```auto
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast(
        tokenizer_file="tokenizer.json",
        bos_token="<s>",
        eos_token="</s>",
        unk_token="<unk>",
        pad_token="<pad>",
    )

```

It gives me errors saying `Using sep_token, but it is not set yet. Using cls_token, but it is not set yet. Using mask_token, but it is not set yet.` However, when I was training the tokenizer or doing the postprocessing, no cls or sep or mask token was involved… I wonder whether anyone has encountered similar problems before?

However, by eyeballing the encoded and decoded results, it looks fine,

```auto
In [20]: tokenizer(
    ...: [["Hello, y'all!", "How are you 😁 ?"], ["Hello to you too!", "I'm fine, thank you!"]]
    ...: )
Out[20]: {'input_ids': [[0, 27193, 15, 92, 10, 5067, 4, 2, 0, 7927, 5082, 6189, 3, 34, 2], [0, 27193, 5000, 6189, 6566, 4, 2, 0, 44, 10, 80, 10447, 15, 23846, 6189, 4, 2]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}

In [21]: tokenizer.decode([0, 27193, 15, 92, 10, 5067, 4, 2, 0, 7927, 5082, 6189, 3, 34, 2])
Out[21]: "<s> Hello, y'all! </s> <s> How are you <unk>? </s>"

```

Thank you so much and appreciate for any help!