# Help defining tokenizer

**URL:** <https://discuss.huggingface.co/t/help-defining-tokenizer/38091>\
**Category:** 🤗Tokenizers\
**Created:** [April 28, 2023, 11:33pm UTC](https://discuss.huggingface.co/t/help-defining-tokenizer/38091 "2023-04-28T23:33:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![surya-narayanan](https://avatars.discourse-cdn.com/v4/letter/s/d2c977/32.png) [@surya-narayanan](https://discuss.huggingface.co/u/surya-narayanan)\
**Post date:** [April 28, 2023, 11:33pm UTC](https://discuss.huggingface.co/t/help-defining-tokenizer/38091/1 "2023-04-28T23:33:28Z")

</div>

Im just curious- i want to train multiple models on the same dataloader, kinda similar to vision- is there any way to train a new tokenizer that’s not specific to a model, such that I can run the following workflow?

```auto

dataset = load_dataset('wiki')
model1 = AutoModelForMaskedLM.from_pretrained("model1name")
model2 = AutoModelForMaskedLM.from_pretrained("model2name")

### help me with code to tokenize the dataset here
tokenizer = (...) # I would have done a from_pretained here, but am not sure what to do, since model1 and model2 might have different tokenizers

def tokenize_function(examples):
    return tokenizer.encode(examples["text"], padding="max_length", truncation=True)

dataset = dataset.map(tokenize_function, batched=True)

dataloader = Dataloader(dataset)
####

for x in dataloader: 
  y1 = model1(x) 
  y2 = model2(x)

```