# How to structure labels for token classification?

**URL:** <https://discuss.huggingface.co/t/how-to-structure-labels-for-token-classification/1216>\
**Category:** 🤗Transformers\
**Created:** [September 22, 2020, 9:03am UTC](https://discuss.huggingface.co/t/how-to-structure-labels-for-token-classification/1216 "2020-09-22T09:03:01Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![jwa018](https://avatars.discourse-cdn.com/v4/letter/j/9de0a6/32.png) [@jwa018](https://discuss.huggingface.co/u/jwa018)\
**Post date:** [September 22, 2020, 9:03am UTC](https://discuss.huggingface.co/t/how-to-structure-labels-for-token-classification/1216/1 "2020-09-22T09:03:01Z")

</div>

The documentation for the label parameter for BertForTokenClassification says that

> Indices should be in `[0, ..., config.num_labels - 1]`

But BertConfig doesn’t have a _num\_labels_ parameter as far as I can tell, so what is this config.num\_labels argument?

Also, in [this](https://huggingface.co/transformers/master/custom_datasets.html) tutorial it says that we can set the labels we want the model to ignore, to -100. If that is correct, why doesn’t the documentation for BertForTokenClassification mention it? Maybe it’s not correct, because when I make my labels like this, I get the error

> /opt/conda/conda-bld/pytorch\_1591914880026/work/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn\_ClassNLLCriterion\_updateOutput\_kernel: block: [0,0,0], thread: [27,0,0] Assertion `t >= 0 && t < n_classes` failed

which indicates to me that I can not have labels with values outside the interval [0, n\_classes].

What I have are labels for which class each wordpiece belongs to, after tokenization. Then I add the special tokens and padding, and I’m setting labels for the special tokens to -100. So for example if I want a sequence length of 10, and I want to classify wordpieces with an ‘o’ in them as class 1, and wordpieces with a ‘p’ in them as class 2, I would have, for the sentence “`Oh, that school is pretty cool`”:

Tokens: [‘oh’, ‘,’, ‘that’, ‘school’, ‘is’, ‘pretty’, ‘cool’]  
With special tokens: [‘[CLS]’, ‘oh’, ‘,’, ‘that’, ‘school’, ‘is’, ‘pretty’, ‘cool’, ‘[SEP]’, ‘[PAD]’]  
Labels: [-100, 1, 0, 0, 1, 0, 2, 0, -100, -100]

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [September 22, 2020, 4:45pm UTC](https://discuss.huggingface.co/t/how-to-structure-labels-for-token-classification/1216/2 "2020-09-22T16:45:45Z")

</div>

You should be able to do something like this:

```python
config = AutoConfig.from_pretrained("bert-base-cased", num_labels=3)
model = AutoModel.from_pretrained("bert-base-cased", config=config)

```

Note that in your example you have three possible labels: with o, with p, **and with neither**. If you set num\_labels to 2, you will have gotten the error that you described.

-100 is the default ignore index for [NLLLoss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss). When a target item has this index, it will be ignored from loss computation.

---

<div class="post-metadata">

**Author:** ![jwa018](https://avatars.discourse-cdn.com/v4/letter/j/9de0a6/32.png) [@jwa018](https://discuss.huggingface.co/u/jwa018)\
**Post date:** [October 8, 2020, 7:59am UTC](https://discuss.huggingface.co/t/how-to-structure-labels-for-token-classification/1216/3 "2020-10-08T07:59:46Z")

</div>

Thank you, this worked. Do you know if it is correct to have -100 as the label for the CLS and SEP tokens when doing token classification? As I understand it, these tokens are required, even if I don’t really use them for anything, since that’s how BERT was pretrained, but I’m not sure how to label them. Maybe they should have the label 0 instead, like the rest of the tokens that don’t belong to any of the other classes, and only the PAD tokens should have the label -100?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [October 9, 2020, 6:42pm UTC](https://discuss.huggingface.co/t/how-to-structure-labels-for-token-classification/1216/4 "2020-10-09T18:42:49Z")

</div>

I’d set all special tokens to - 100. You are not interested in the performance of your model on those tokens.

---

<div class="post-metadata">

**Author:** ![enesz](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/enesz/32/2592_2.png) [@enesz](https://discuss.huggingface.co/u/enesz)\
**Post date:** [August 29, 2021, 11:56am UTC](https://discuss.huggingface.co/t/how-to-structure-labels-for-token-classification/1216/5 "2021-08-29T11:56:28Z")

</div>

And how to label padding? Should it also be -100?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [August 29, 2021, 7:38pm UTC](https://discuss.huggingface.co/t/how-to-structure-labels-for-token-classification/1216/6 "2021-08-29T19:38:00Z")

</div>

Yes, padding is also typically ignored.