# Tokenization in a NER context

**URL:** <https://discuss.huggingface.co/t/tokenization-in-a-ner-context/5635>\
**Category:** 🤗Tokenizers\
**Created:** [April 20, 2021, 1:13pm UTC](https://discuss.huggingface.co/t/tokenization-in-a-ner-context/5635 "2021-04-20T13:13:06Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Thrix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thrix/32/1533_2.png) [@Thrix](https://discuss.huggingface.co/u/Thrix)\
**Post date:** [April 20, 2021, 1:13pm UTC](https://discuss.huggingface.co/t/tokenization-in-a-ner-context/5635/1 "2021-04-20T13:13:06Z")

</div>

Hello everyone, I am trying to understand how to use the tokenizers in a NER context.

Basically, I have a text corpus with entities annotations, usually in IOB format [1], which can be seen as a mapping f: word → tag (annotators are working on a non-tokenized text and we ask them to annotate entire words).

When I am using any modern tokenizer, basically, I will get several tokens for a single word (for instance “huggingface” might produce something like [“hugging#”, “face”]). I need to transfer the original annotations to each token in order to have a new labelling fonction g: token → tag

E.g. what I have in input

```auto
text = "Huggingface is amazing"
labels = [B_org, O, O]"

```

what I need to produce if the tokenizer output is `["Hugging#", "face", "is", "amazin"]` is

````auto
labels_per_tokens = [B_org, I_org, O, O]
```

To do so I need to backtrack for every token produced by the tokenizer what is the original word / annotation that I got in input but it seems not so easy to do (especially with [UNK] tokens). Am I missing something obvious ? Are there some good practice or solution to my problem ?

Thanks a lot for your help !

[1] https://en.wikipedia.org/wiki/Inside%E2%80%93outside%E2%80%93beginning_(tagging)
````

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [April 20, 2021, 1:46pm UTC](https://discuss.huggingface.co/t/tokenization-in-a-ner-context/5635/2 "2021-04-20T13:46:14Z")

</div>

hey @Thrix you can see how to align the NER tags and tokens in the `tokenize_and_align_labels` function in this tutorial: [https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token\_classification.ipynb#scrollTo=n9qywopnIrJH](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/token_classification.ipynb#scrollTo=n9qywopnIrJH)

---

<div class="post-metadata">

**Author:** ![Thrix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thrix/32/1533_2.png) [@Thrix](https://discuss.huggingface.co/u/Thrix)\
**Post date:** [April 20, 2021, 2:04pm UTC](https://discuss.huggingface.co/t/tokenization-in-a-ner-context/5635/3 "2021-04-20T14:04:16Z")

</div>

Thanks a lot for the link ! Sounds exactly what I’m looking for ; I will check the tutorial !

---

<div class="post-metadata">

**Author:** ![sbmaruf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sbmaruf/32/1528_2.png) [@sbmaruf](https://discuss.huggingface.co/u/sbmaruf)\
**Post date:** [April 29, 2021, 4:43pm UTC](https://discuss.huggingface.co/t/tokenization-in-a-ner-context/5635/4 "2021-04-29T16:43:03Z")

</div>

For a word that is divided into multiple tokens by bpe or sentencepiece like model, you use the first token as your reference token that you want to predict. Since all the tokens are connected via self-attention you won’t have the problem not predicting the rest of the bpe tokens of a word. In PyTorch, you can ignore [computing loss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) (see _ignore\_index_ argument) of those tokens by providing -100 as a label to those tokens (life is so easy with pytorch 😉 ).

---

<div class="post-metadata">

**Author:** ![Thrix](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thrix/32/1533_2.png) [@Thrix](https://discuss.huggingface.co/u/Thrix)\
**Post date:** [May 7, 2021, 12:13pm UTC](https://discuss.huggingface.co/t/tokenization-in-a-ner-context/5635/5 "2021-05-07T12:13:49Z")

</div>

Thanks for the trick ! Indeed, that’s also a very reasonable way to go

---

<div class="post-metadata">

**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\
**Post date:** [August 11, 2021, 7:07am UTC](https://discuss.huggingface.co/t/tokenization-in-a-ner-context/5635/6 "2021-08-11T07:07:28Z")

</div>

@lewtun - Just wanted to know, how to align multiple sub words into individual word in case of SentencePiece tokenizer like Albert, for NER?

For example,

```auto
My name is Abraham George and I am living in Philadelphia" .

```

Now this will be split into sub words, and predictions happens per sub word. Now how we align it back to the `individual words`.

@sgugger