# KeyError: 'loss' when fine-tuning a Transformer model

**URL:** <https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948>\
**Category:** Beginners\
**Created:** [February 20, 2022, 8:44pm UTC](https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948 "2022-02-20T20:44:38Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ryedida](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@ryedida](https://discuss.huggingface.co/u/ryedida)\
**Post date:** [February 20, 2022, 8:44pm UTC](https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948/1 "2022-02-20T20:44:38Z")

</div>

I am trying to fine tune a transformer model on my own unlabeled corpus of text. My code for doing this is:

```auto
from datasets import load_dataset
from transformers import BertTokenizerFast
from transformers import AutoModel
from transformers import TrainingArguments
from transformers import Trainer
import glob
import os

base_path = '../data/'
model_name = 'bert-base-uncased'
max_length = 512
checkpoints_dir = 'checkpoints'

if not os.path.exists(checkpoints_dir):
    os.mkdir(checkpoints_dir)

tokenizer = BertTokenizerFast.from_pretrained(model_name, do_lower_case=True)

def tokenize_function(examples):
    return tokenizer(examples['text'], padding='max_length', truncation=True, max_length=max_length)

dataset = load_dataset('text',
        data_files={
            'train': f'{base_path}train.txt',
            'test': f'{base_path}test.txt',
            'validation': f'{base_path}valid.txt'
        }
)

print('Tokenizing data. This may take a while...')
tokenized_dataset = dataset.map(tokenize_function, batched=True)
train_dataset = tokenized_dataset['train']
eval_dataset = tokenized_dataset['test']

model = AutoModel.from_pretrained(model_name)

training_args = TrainingArguments(checkpoints_dir)

trainer = Trainer(model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset)
trainer.train()

```

However, I get `KeyError: 'loss'` when running the code at `trainer.train()`. How do I fix this?

---

<div class="post-metadata">

**Author:** ![beneyal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beneyal/32/27988_2.png) [@beneyal](https://discuss.huggingface.co/u/beneyal)\
**Post date:** [February 20, 2022, 9:25pm UTC](https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948/2 "2022-02-20T21:25:23Z")

</div>

Hello! 👋

Make sure your dataset has a `"labels"` column, otherwise the `Trainer` won’t recognize the labels and won’t calculate a loss.

---

<div class="post-metadata">

**Author:** ![ryedida](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@ryedida](https://discuss.huggingface.co/u/ryedida)\
**Post date:** [February 20, 2022, 9:36pm UTC](https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948/3 "2022-02-20T21:36:54Z")

</div>

Thanks for the swift reply, @beneyal! I don’t have labels in my data, since my end goal is to simply fine-tune the weights and use it for multiple tasks (but primarily for generating embeddings). How can I achieve that?

---

<div class="post-metadata">

**Author:** ![beneyal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beneyal/32/27988_2.png) [@beneyal](https://discuss.huggingface.co/u/beneyal)\
**Post date:** [February 20, 2022, 9:43pm UTC](https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948/4 "2022-02-20T21:43:35Z")

</div>

No problem, you’re very welcome! 🙂

Even if you just want to fine-tune the embedding weights, you need some kind of labelled dataset so that a loss will be calculated and the errors propagated. If you don’t have a labelled dataset for some specific downstream task, you can always go for Masked Language Modeling or Causal Language Modeling pre-training routines.

---

<div class="post-metadata">

**Author:** ![ryedida](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@ryedida](https://discuss.huggingface.co/u/ryedida)\
**Post date:** [February 21, 2022, 5:30pm UTC](https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948/5 "2022-02-21T17:30:05Z")

</div>

Thanks a lot! Using the MLM tutorials helped me fix this!

---

<div class="post-metadata">

**Author:** ![vinrok](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vinrok/32/4368_2.png) [@vinrok](https://discuss.huggingface.co/u/vinrok)\
**Post date:** [July 11, 2022, 11:10am UTC](https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948/6 "2022-07-11T11:10:21Z")

</div>

Hi @ryedida,

Could you please guide me on this? I am also facing the same issue and I am using MLM method.

Here is the link - [Google Colab](https://colab.research.google.com/drive/1LhnRp-tvxUEm3etxod3zb90JCHnSyFqc?usp=sharing)

Thanks.

---

<div class="post-metadata">

**Author:** ![ryedida](https://avatars.discourse-cdn.com/v4/letter/r/c57346/32.png) [@ryedida](https://discuss.huggingface.co/u/ryedida)\
**Post date:** [July 12, 2022, 2:36am UTC](https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948/7 "2022-07-12T02:36:10Z")

</div>

[Here is my final code](https://github.com/yrahul3910/se-language-models/blob/master/train_lm_hf.py); hopefully, this helps you out.

---

<div class="post-metadata">

**Author:** ![vinrok](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vinrok/32/4368_2.png) [@vinrok](https://discuss.huggingface.co/u/vinrok)\
**Post date:** [July 12, 2022, 8:12pm UTC](https://discuss.huggingface.co/t/keyerror-loss-when-fine-tuning-a-transformer-model/14948/8 "2022-07-12T20:12:28Z")

</div>

Hi @ryedida,

I am really grateful for your timely response but what to do after that?

As I want to perform semantic search based on the embeddings obtained by training model on my custom dataset.

Kindly guide me.

Thanks.