# Tokenizer from scratch Error TypeError: Can't convert None to PyString

**URL:** <https://discuss.huggingface.co/t/tokenizer-from-scratch-error-typeerror-cant-convert-none-to-pystring/28395>\
**Category:** Beginners\
**Created:** [December 25, 2022, 8:31am UTC](https://discuss.huggingface.co/t/tokenizer-from-scratch-error-typeerror-cant-convert-none-to-pystring/28395 "2022-12-25T08:31:55Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![drsis](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@drsis](https://discuss.huggingface.co/u/drsis)\
**Post date:** [December 25, 2022, 8:31am UTC](https://discuss.huggingface.co/t/tokenizer-from-scratch-error-typeerror-cant-convert-none-to-pystring/28395/1 "2022-12-25T08:31:55Z")

</div>

Hi,

I tried to train a tokenizer from scratch with my own created dataset.  
The dataset has the following structure:

```auto
DatasetDict({
    train: Dataset({
        features: ['slug', 'date', 'type', 'content', 'word_count', ' __index_level_0__'],
        num_rows: 3048752
    })
})

```

I now tried to train a ByteLevelBPETokenizer from scratch, but I always get the Error “TypeError: Can’t convert None to pystring”

```auto
from transformers import AutoTokenizer
from datasets import load_dataset
from tokenizers import ByteLevelBPETokenizer
from tqdm import tqdm

raw_dataset = load_dataset('drsis/law_de_courtdecisions')

def get_training_corpus():
    dataset = raw_dataset['train']
    for start_idx in range(0, len(dataset), 1000):
        samples = dataset[start_idx : start_idx + 1000]
        yield samples['content']

## Initialize an empty tokenizer
tokenizer = ByteLevelBPETokenizer(add_prefix_space=True)

# Train the tokenizer on the corpus using the generator get_training_corpus as a source
tokenizer.train_from_iterator(get_training_corpus(), 
            vocab_size=52_000, 
            min_frequency=2, 
            special_tokens=["<s>", "<pad>", "</s>", "<unk>", "<mask>"],
            show_progress=True)

```

Has somebody an idea what the issue is. I tried to go over the whole content key of the dataset and attempted to find out if there is some None data, but this was not the case.

I would appreciate it if somebody could help me out.

Thanks

---

<div class="post-metadata">

**Author:** ![drsis](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@drsis](https://discuss.huggingface.co/u/drsis)\
**Post date:** [December 26, 2022, 8:04am UTC](https://discuss.huggingface.co/t/tokenizer-from-scratch-error-typeerror-cant-convert-none-to-pystring/28395/2 "2022-12-26T08:04:52Z")

</div>

Found the error by myself.  
There are 171 empty elements in the dataset which caused the TypeError.