# Custom Dataset with Custom Tokenizer

**URL:** <https://discuss.huggingface.co/t/custom-dataset-with-custom-tokenizer/7100>\
**Category:** 🤗Datasets\
**Created:** [June 23, 2021, 12:18pm UTC](https://discuss.huggingface.co/t/custom-dataset-with-custom-tokenizer/7100 "2021-06-23T12:18:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![isarth](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/isarth/32/13332_2.png) [@isarth](https://discuss.huggingface.co/u/isarth)\
**Post date:** [June 23, 2021, 12:18pm UTC](https://discuss.huggingface.co/t/custom-dataset-with-custom-tokenizer/7100/1 "2021-06-23T12:18:31Z")

</div>

I trained a BPE tokenizer using the wiki-text and now I’m trying to use this tokenizer on a custom dataset from a csv file. What I want to achieve is additional of feature columns in my dataset. But the dataset.map is giving error.

 ![Screenshot 2021-06-23 at 5.47.51 PM](https://us1.discourse-cdn.com/hellohellohello/original/2X/9/99822a36fe1d039de1d8f2b675f981bca377d8fa.png)

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [June 23, 2021, 12:26pm UTC](https://discuss.huggingface.co/t/custom-dataset-with-custom-tokenizer/7100/2 "2021-06-23T12:26:16Z")

</div>

You should just use the tokenizer ` __call__ `: `tokenizer(example["text"])`.

---

<div class="post-metadata">

**Author:** ![isarth](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/isarth/32/13332_2.png) [@isarth](https://discuss.huggingface.co/u/isarth)\
**Post date:** [June 23, 2021, 12:38pm UTC](https://discuss.huggingface.co/t/custom-dataset-with-custom-tokenizer/7100/3 "2021-06-23T12:38:23Z")

</div>

When I train the tokenizer using this [Quicktour — tokenizers documentation](https://huggingface.co/docs/tokenizers/python/latest/quicktour.html)  
The call function is not implemented

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [June 23, 2021, 12:50pm UTC](https://discuss.huggingface.co/t/custom-dataset-with-custom-tokenizer/7100/4 "2021-06-23T12:50:41Z")

</div>

Oh, you should wrap your tokenizer in a `PreTrainedTokenizerFact` from the Transformers library (you can just pass your tokenizer with the `tokenizer_object` keyword argument).