# Finetuning from multiclass to mutlilabel

**URL:** <https://discuss.huggingface.co/t/finetuning-from-multiclass-to-mutlilabel/9658>\
**Category:** Intermediate\
**Created:** [September 1, 2021, 8:19am UTC](https://discuss.huggingface.co/t/finetuning-from-multiclass-to-mutlilabel/9658 "2021-09-01T08:19:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 1, 2021, 8:19am UTC](https://discuss.huggingface.co/t/finetuning-from-multiclass-to-mutlilabel/9658/1 "2021-09-01T08:19:13Z")

</div>

Hello,

I got a specific classification task where I finetuned a pretrained BERT Model on a specific task concerning customer reviews (classify a text as “customer service text”, “user experience” etc.):

```auto
num_labels_cla = 8
model_name_cla = "bert-base-german-dbmdz-uncased"
batch_size_cla = 32

model = AutoModelForSequenceClassification.from_pretrained(model_name_cla, num_labels=num_labels_cla)
tokenizer = AutoTokenizer.from_pretrained(model_name_cla)

```

As you can see I got 8 distinct classes. My finetuned classification model scores pretty well with unseen data with a f1 score of 80.1. However, it is possible that one text belongs to 2 different classes. My question now is how I have to change my code achieve that? I already transformed my target variable with MultiLabelBinarizer such that my target variable looks like this:

```auto
       [0, 0, 0, 1, 0, 0, 0, 0],
       [1, 0, 0, 0, 0, 0, 0, 0],
       [1, 0, 0, 1, 0, 0, 0, 0],

```

I am using the HuggingFace Trainer instance for finetuning.

Cheers

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 1, 2021, 1:18pm UTC](https://discuss.huggingface.co/t/finetuning-from-multiclass-to-mutlilabel/9658/2 "2021-09-01T13:18:51Z")

</div>

You can set the `problem_type` of an `xxxForSequenceClassification` model to `multi_label_classification` when instantiating it, like so:

```auto
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("bert-base-german-dbmz-uncased", problem_type="multi_label_classification", num_labels=num_labels_cla)

```

This ensures that the `BCEWithLogitsLoss` is used instead of the `CrossEntropyLoss`, which is necessary for multi-label classification. You can then fine-tune just like you would do with multi-class classification.

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 1, 2021, 1:46pm UTC](https://discuss.huggingface.co/t/finetuning-from-multiclass-to-mutlilabel/9658/3 "2021-09-01T13:46:04Z")

</div>

Thank you for your reply! Is it correct then that I encode my target like this:

> [@marlon89](#):
>
> ```auto
> [0, 0, 0, 1, 0, 0, 0, 0],
> [1, 0, 0, 0, 0, 0, 0, 0],
> [1, 0, 0, 1, 0, 0, 0, 0],
> 
> ```

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [September 1, 2021, 1:53pm UTC](https://discuss.huggingface.co/t/finetuning-from-multiclass-to-mutlilabel/9658/4 "2021-09-01T13:53:16Z")

</div>

Yes 🙂 as you can see in the [code](https://github.com/huggingface/transformers/blob/c1b20e42f5f50704434557523defae05e2854125/src/transformers/models/bert/modeling_bert.py#L1560-L1562), the loss is computed as follows:

```auto
elif self.config.problem_type == "multi_label_classification":
      loss_fct = BCEWithLogitsLoss()
      loss = loss_fct(logits, labels)

```

The logits will be of shape `(batch_size, num_labels)`. The [docs](https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html) of PyTorch’s `BCEWithLogitsLoss` indicates that the `labels` should have the same shape. So that’s indeed correct.

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 1, 2021, 1:54pm UTC](https://discuss.huggingface.co/t/finetuning-from-multiclass-to-mutlilabel/9658/5 "2021-09-01T13:54:00Z")

</div>

Thank you so much for your support! 🙂