# Loss error for bert token classifier

**URL:** <https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460>\
**Category:** Beginners\
**Created:** [December 2, 2021, 3:17pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460 "2021-12-02T15:17:00Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![AgelZin](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@AgelZin](https://discuss.huggingface.co/u/AgelZin)\
**Post date:** [December 2, 2021, 3:17pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/1 "2021-12-02T15:17:00Z")

</div>

So i am doing my first berttoken classifier. I am using a german polyglot dataset meaning tokenised words and lists of ner labels.  
a row is [‘word1’,‘word2’…] [‘ORG’,‘LOC’…]  
This is my code  
`tokenizer = BertTokenizer.from_pretrained('bert-base-german-cased')`  
`encoded_dataset = [tokenizer(item['words'], is_split_into_words=True,return_tensors="pt", padding='max_length', truncation=True, max_length=128) for item in dataset_1]`  
`model = BertForTokenClassification.from_pretrained('bert-base-german-cased', num_labels=1)`

```auto
for item in encoded_dataset:

    for key in item:

        item[key] = torch.squeeze(item[key])

train_set = encoded_dataset[:500]

test_set = encoded_dataset[500:]

```

training\_args = TrainingArguments(

```
num_train_epochs=1,

per_device_train_batch_size=4,

per_device_eval_batch_size=4,

output_dir='results',

logging_dir='logs',

no_cuda=False, # defaults to false anyway, just to be explicit

```

)

```auto

trainer = Trainer(

    model=model,

    tokenizer=tokenizer,

    args=training_args,

    train_dataset=train_set,

)

```

`trainer.train()`

And i am getting key error loss

---

<div class="post-metadata">

**Author:** ![emmakelo](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@emmakelo](https://discuss.huggingface.co/u/emmakelo)\
**Post date:** [December 2, 2021, 3:32pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/2 "2021-12-02T15:32:20Z")

</div>

Could you post the error ?

---

<div class="post-metadata">

**Author:** ![AgelZin](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@AgelZin](https://discuss.huggingface.co/u/AgelZin)\
**Post date:** [December 2, 2021, 5:46pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/3 "2021-12-02T17:46:16Z")

</div>

![error](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/17f88f4161d4d16ce814b4238030f8ba2ae0110d.png)

---

<div class="post-metadata">

**Author:** ![emmakelo](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@emmakelo](https://discuss.huggingface.co/u/emmakelo)\
**Post date:** [December 3, 2021, 8:35am UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/4 "2021-12-03T08:35:59Z")

</div>

The problem seems to be in the Trainer. How is your data encoded ? can you show the shape, type ans how it looks before passing to the trainer ?

your num\_lables = 1 , Are you doing single classification ?

---

<div class="post-metadata">

**Author:** ![emmakelo](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@emmakelo](https://discuss.huggingface.co/u/emmakelo)\
**Post date:** [December 3, 2021, 8:37am UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/5 "2021-12-03T08:37:52Z")

</div>

Try putting num\_train\_epoch to flioat number = 1.0 to see if it works and also check the number of label? if it really 1 label in your training data ?

---

<div class="post-metadata">

**Author:** ![AgelZin](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@AgelZin](https://discuss.huggingface.co/u/AgelZin)\
**Post date:** [December 3, 2021, 2:44pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/6 "2021-12-03T14:44:05Z")

</div>

num of labels was a mistake i changed it to 4 since they are 4 types. I didnt do any further encoding to the data than this code

---

<div class="post-metadata">

**Author:** ![emmakelo](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@emmakelo](https://discuss.huggingface.co/u/emmakelo)\
**Post date:** [December 3, 2021, 3:46pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/7 "2021-12-03T15:46:21Z")

</div>

when you change label does it outputs the same result ?

---

<div class="post-metadata">

**Author:** ![AgelZin](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@AgelZin](https://discuss.huggingface.co/u/AgelZin)\
**Post date:** [December 3, 2021, 3:56pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/8 "2021-12-03T15:56:51Z")

</div>

yes the float number doesnt change it

---

<div class="post-metadata">

**Author:** ![emmakelo](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@emmakelo](https://discuss.huggingface.co/u/emmakelo)\
**Post date:** [December 3, 2021, 4:22pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/9 "2021-12-03T16:22:24Z")

</div>

Could you print dataset\_1 to see how it looks ?

---

<div class="post-metadata">

**Author:** ![AgelZin](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@AgelZin](https://discuss.huggingface.co/u/AgelZin)\
**Post date:** [December 3, 2021, 4:51pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/10 "2021-12-03T16:51:48Z")

</div>

![Capture](https://us1.discourse-cdn.com/hellohellohello/original/2X/b/b6aedaf8d4efa2552688d1d8a93b6fbad58b5a15.png)

---

<div class="post-metadata">

**Author:** ![emmakelo](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@emmakelo](https://discuss.huggingface.co/u/emmakelo)\
**Post date:** [December 3, 2021, 5:14pm UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/11 "2021-12-03T17:14:51Z")

</div>

I think maybe you should change dataset to Dataset type and then rewrite like this :

tokenized\_dataset = dataset\_1.map(lambda x: tokenizer(x[‘words’], is\_split\_into\_words=True,return\_tensors=“pt”, padding=‘max\_length’, truncation=True, max\_length=128)

I think you donot need to loop into dataset\_1 but rather pass the column words dataset\_1[‘words’] directly to the tokenizer or transform to Dataset format. [Datasets — datasets 1.16.1 documentation](https://huggingface.co/docs/datasets/)

---

<div class="post-metadata">

**Author:** ![AgelZin](https://avatars.discourse-cdn.com/v4/letter/a/6bbea6/32.png) [@AgelZin](https://discuss.huggingface.co/u/AgelZin)\
**Post date:** [December 4, 2021, 1:46am UTC](https://discuss.huggingface.co/t/loss-error-for-bert-token-classifier/12460/12 "2021-12-04T01:46:27Z")

</div>

the tokenized dataset didnt work. I think i need to do some label encoding first for ner but not sure how to go about that