# Save tokenizer with argument

**URL:** <https://discuss.huggingface.co/t/save-tokenizer-with-argument/17389>\
**Category:** 🤗Tokenizers\
**Created:** [May 1, 2022, 1:55pm UTC](https://discuss.huggingface.co/t/save-tokenizer-with-argument/17389 "2022-05-01T13:55:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![petarulev](https://avatars.discourse-cdn.com/v4/letter/p/c5a1d2/32.png) [@petarulev](https://discuss.huggingface.co/u/petarulev)\
**Post date:** [May 1, 2022, 1:55pm UTC](https://discuss.huggingface.co/t/save-tokenizer-with-argument/17389/1 "2022-05-01T13:55:14Z")

</div>

I am training my huggingface tokenizer on my own corpora, and I want to save it with a preprocessing step. That is, if I pass some text to it, I want it to apply the preprocessing and then tokenize the text, instead of explicitly preprocessing it before that. A good example is BERTweet: [GitHub - VinAIResearch/BERTweet: BERTweet: A pre-trained language model for English Tweets (EMNLP-2020)](https://github.com/VinAIResearch/BERTweet) and their `tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base", normalization=True)` (here normalization=True indicates that the input will be preprocessed according to some function). I want the same to apply when I train a tokenizer with a custom preprocessing function. My code is:

```
from pathlib import Path
from tokenizers import ByteLevelBPETokenizer

def preprocess(text):
    return text

paths = [str(x) for x in Path('data').glob('*.txt')]

tokenizer = ByteLevelBPETokenizer()

tokenizer.train(files=paths, vocab_size=50_000, min_frequency=2,
                special_tokens=['<s>', '<pad>', '</s>', '<unk>', '<mask>'])

tokenizer.save_model('CustomBertTokenizer')

```

Now, when I load the tokenizer:

```
from transformers import RobertaTokenizerFast
sentence = 'Hey'
tokenizer = RobertaTokenizerFast.from_pretrained('CustomBertTokenizer')
tokenizer(sentence)

```

I want `sentence` to be preprocessed with the `preprocess` function, and then tokenized. So I want to pass like an argument : preprocessing=True, or something like that. How can I do it?

---

<div class="post-metadata">

**Author:** ![petarulev](https://avatars.discourse-cdn.com/v4/letter/p/c5a1d2/32.png) [@petarulev](https://discuss.huggingface.co/u/petarulev)\
**Post date:** [May 8, 2022, 2:35pm UTC](https://discuss.huggingface.co/t/save-tokenizer-with-argument/17389/2 "2022-05-08T14:35:21Z")

</div>

Any ideas guys?

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](https://discuss.huggingface.co/u/lianghsun)\
**Post date:** [October 26, 2022, 9:43am UTC](https://discuss.huggingface.co/t/save-tokenizer-with-argument/17389/3 "2022-10-26T09:43:27Z")

</div>

Hi @petarulev , whatever you want to do is:

> _Preprocess_ → _Tokenization_ → _Model_

Then, I’ll suggest you using [HF Datasets](https://huggingface.co/docs/datasets/process#map) at first, so the code should be:

```auto
# Suppose I have a dataset in .txt format, and the context is:
# 
# C1CCCC1\n
# C1NNCC1\n
# ...and so on

# Step1: load datasets with HF Datasets
from datasets import load_dataset
ds = load_dataset(
    "text", data_files={'train': ['path/to/data_1', ['path/to/data_2']}
)

# Step2: use .map() method
def preprocess(example):
    # You can find out that each line in dataset file contain
    # unwanted breaking line symbol '\n', so i decide to remove it
    return example['text'] = example['text'].rstrip()

processed_ds = ds.map(preprocess, num_proc=4) # set `num_proc` to speed up!

# Step3: tokenization
tokenizer = ...

```

And this should be the way you can try. 🤗