# NLP dataset for ByteLevelTokenizer Training

**URL:** <https://discuss.huggingface.co/t/nlp-dataset-for-byteleveltokenizer-training/3653>\
**Category:** 🤗Datasets\
**Created:** [February 15, 2021, 9:47pm UTC](https://discuss.huggingface.co/t/nlp-dataset-for-byteleveltokenizer-training/3653 "2021-02-15T21:47:47Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![tillfurger](https://avatars.discourse-cdn.com/v4/letter/t/6bbea6/32.png) [@tillfurger](https://discuss.huggingface.co/u/tillfurger)\
**Post date:** [February 15, 2021, 9:47pm UTC](https://discuss.huggingface.co/t/nlp-dataset-for-byteleveltokenizer-training/3653/1 "2021-02-15T21:47:48Z")

</div>

Hi I would like to train my own `ByteLevelBPETokenizer` using an nlp dataset.

```
tokenizer = ByteLevelBPETokenizer()

tokenizer.train(files=???, vocab_size=52000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

```

The dataset is from:

```
from datasets import load_dataset
dataset = load_dataset('wikicorpus', 'raw_en')

```

How can I process this dataset to input it in the `tokenizer.train()` function?

Thanks

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [February 16, 2021, 2:10pm UTC](https://discuss.huggingface.co/t/nlp-dataset-for-byteleveltokenizer-training/3653/2 "2021-02-16T14:10:39Z")

</div>

You can take a look at the example script here:

> <https://github.com/huggingface/tokenizers/blob/master/bindings/python/examples/train_with_datasets.py>