# 1 line code for NER data set preparation using tokenizer library!

**URL:** <https://discuss.huggingface.co/t/1-line-code-for-ner-data-set-preparation-using-tokenizer-library/22816>\
**Category:** 🤗Tokenizers\
**Created:** [September 9, 2022, 3:40am UTC](https://discuss.huggingface.co/t/1-line-code-for-ner-data-set-preparation-using-tokenizer-library/22816 "2022-09-09T03:40:54Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Imran1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imran1/32/18779_2.png) [@Imran1](https://discuss.huggingface.co/u/Imran1)\
**Post date:** [September 9, 2022, 3:40am UTC](https://discuss.huggingface.co/t/1-line-code-for-ner-data-set-preparation-using-tokenizer-library/22816/1 "2022-09-09T03:40:55Z")

</div>

you can see this code is hard to read and understand. Is there any easy way in which we can simplify 15 line of code to 1 line???

```auto
def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True)

    labels = []
    for i, label in enumerate(examples[f"{task}_tags"]):
        word_ids = tokenized_inputs.word_ids(batch_index=i)
        previous_word_idx = None
        label_ids = []
        for word_idx in word_ids:
            # Special tokens have a word id that is None. We set the label to -100 so they are automatically
            # ignored in the loss function.
            if word_idx is None:
                label_ids.append(-100)
            # We set the label for the first token of each word.
            elif word_idx != previous_word_idx:
                label_ids.append(label[word_idx])
            # For the other tokens in a word, we set the label to either the current label or -100, depending on
            # the label_all_tokens flag.
            else:
                label_ids.append(label[word_idx] if label_all_tokens else -100)
            previous_word_idx = word_idx

        labels.append(label_ids)

    tokenized_inputs["labels"] = labels
    return tokenized_inputs

```