# Load custom pretrained tokenizer

**URL:** <https://discuss.huggingface.co/t/load-custom-pretrained-tokenizer/11148>\
**Category:** 🤗Tokenizers\
**Created:** [October 28, 2021, 1:36am UTC](https://discuss.huggingface.co/t/load-custom-pretrained-tokenizer/11148 "2021-10-28T01:36:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbmaxwell](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@jbmaxwell](https://discuss.huggingface.co/u/jbmaxwell)\
**Post date:** [October 28, 2021, 1:36am UTC](https://discuss.huggingface.co/t/load-custom-pretrained-tokenizer/11148/1 "2021-10-28T01:36:00Z")

</div>

I’m trying to run BigBird on my dataset but I’m hitting an error trying to load my custom/saved tokenizer.  
I train the tokenizer using:

```auto
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace

tokenizer = Tokenizer(BPE())
tokenizer.pre_tokenizer = Whitespace()
paths = ['./content/test.txt', './content/train.txt']
from tokenizers.trainers import BpeTrainer

trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
tokenizer.train(files=paths, trainer=trainer)

# Save files to disk
tokenizer.model.save("./tokenizer/bigbird_full")
tokenizer.save("./tokenizer/bigbird_full/config.json")

```

Then I try to run `run_mlm.py` with:

```auto
os.system(
    f"python run_mlm.py \
       --model_type 'big_bird' \
       --config_name './models/bigbird_full/config.json' \
       --tokenizer_name './tokenizer/bigbird_full' \
       --train_file './content/train.txt' \
       --output_dir './{out_dir}' \
       --do_train \
       --num_train_epochs 1 \
       --overwrite_output_dir"
)

```

It fails with the error:

```auto
OSError: Can't load tokenizer for './tokenizer/bigbird_full'. Make sure that:

- './tokenizer/bigbird_full' is a correct model identifier listed on 'https://huggingface.co/models'
  (make sure './tokenizer/bigbird_full' is not a path to a local directory with something else, in that case)

- or './tokenizer/bigbird_full' is the correct path to a directory containing relevant tokenizer files

- or 'main' is a valid git identifier (branch name, a tag name, or a commit id) that exists for this model name as listed on its model page on 'https://huggingface.co/models'

```

My tokenizer’s directory contains: `config.json`, `merges.txt`, `vocab.json`

What is the error trying to tell me? Or rather, what are “relevant tokenizer files” if not these?

PS – It’s worth noting that, for some reason, I have to manually add `"model_type":"big_bird"` to my saved `config.json`, otherwise I get an error telling me basically exactly that… (i.e., it needs that key/val).