# Most efficient multi-label classifier?

**URL:** <https://discuss.huggingface.co/t/most-efficient-multi-label-classifier/9296>\
**Category:** Beginners\
**Created:** [August 17, 2021, 3:45pm UTC](https://discuss.huggingface.co/t/most-efficient-multi-label-classifier/9296 "2021-08-17T15:45:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![qedgary](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/qedgary/32/2491_2.png) [@qedgary](https://discuss.huggingface.co/u/qedgary)\
**Post date:** [August 17, 2021, 3:45pm UTC](https://discuss.huggingface.co/t/most-efficient-multi-label-classifier/9296/1 "2021-08-17T15:45:40Z")

</div>

#### Background

I’m trying to train a model in Tensorflow to classify text according to a fixed set of 5 labels. For example, let’s say I feed my model the following text:

> “my advice is that you go ahead with your plans to learn Python, because its syntax is easy for beginners. It’s also great for snake lovers like me!”

After sniffing the text, the model would, ideally, report back how much the text matches my pre-defined labels:

```auto
       Label Prediction
-------------------- ----------
programming_advice 0.99
advice_for_beginners 0.91
cooking_advice 0.11
health_advice 0.10
not_advice 0.01

```

#### My question

_ **What is the most efficient way to build such a classifier?** _ I’ve seen several options to do this, but I’m not sure which one would be best:

1. _Fine-tune five different binary classifiers, since there are five labels_… but this would take forever to train, so I assume there must be a better way.
2. _Make a model with a transformer only, and train it._
  - [Seen in this forum question](https://discuss.huggingface.co/t/mismatched-target-and-input-size-for-bce-using-multi-label-classification/8706)

3. _Make a model with a transformer plus my own Dense layers, and train it_.
  - [Seen in this sample notebook](https://github.com/DhavalTaunk08/NLP_scripts/blob/master/Transformers_multilabel_distilbert.ipynb), which was linked in the transformers documentation.

4. _Make a model with a transformer plus my own Dense layers—but freeze the transformer as-is, and only train the Dense layers._
  - Freezing is a common practice with pre-trained computer vision models; I don’t know whether it’s also good practice for NLP too.

I would be grateful for any suggestions on which of 1-4 works best. I’m still rather new around here, but the Huggingface community is extremely welcoming and helpful, and I appreciate being here! A big thanks for anybody who can help give me some pointers.

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [August 23, 2021, 2:54pm UTC](https://discuss.huggingface.co/t/most-efficient-multi-label-classifier/9296/2 "2021-08-23T14:54:25Z")

</div>

I am facing the same problem and as none replied yet I wanted to ask if you got any updates/new thoughts on this? Cheers

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [August 23, 2021, 6:51pm UTC](https://discuss.huggingface.co/t/most-efficient-multi-label-classifier/9296/3 "2021-08-23T18:51:13Z")

</div>

Hi,

Option 2 is indeed the best. To train a multi-label classifier, you can use an `xxxForSequenceClassification` model (which is a Transformer encoder with a linear layer on top), and set the `problem_type` attribute of the configuration to `multi_label_classification`. For example, if you want to use BERT, you can do it as follows:

```auto
from transformers import BertForSequenceClassification

model = BertForSequenceClassification.from_pretrained("bert-base-uncased", problem_type="multi_label_classification")

```

As you can see in the [code](https://github.com/huggingface/transformers/blob/2772d3e79d66925cf4adeaffd8be610f0ab177b6/src/transformers/models/bert/modeling_bert.py#L1561-L1563), it will use the BCE (binary cross-entropy) loss.

Note that if the number of labels you have is \> 2, you also need to specify `num_labels=...` when calling the `.from_pretrained()` method.

---

<div class="post-metadata">

**Author:** ![vettukal](https://avatars.discourse-cdn.com/v4/letter/v/8e8cbc/32.png) [@vettukal](https://discuss.huggingface.co/u/vettukal)\
**Post date:** [September 1, 2022, 11:18pm UTC](https://discuss.huggingface.co/t/most-efficient-multi-label-classifier/9296/4 "2022-09-01T23:18:41Z")

</div>

Below is the code sample that I managed to make it work by using the [multi\_label\_classification](https://discuss.huggingface.co/t/mismatched-target-and-input-size-for-bce-using-multi-label-classification/8706/2) question

```auto

import torch
from torch.utils.data.dataset import Dataset

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Example data. 
# In reality, the strings are usually longer and there are 11 possible classes
texts = [
    "This is the first sentence.",
    "This is the second sentence.",
    "This is another sentence.",
    "Finally, the last sentence.",
]

labels = [
    [0.99, 0.91, 0.11, 0.10, 0.01],
    [0.89, 0.51, 0.01, 0.10, 0.01],
    [0.39, 0.21, 0.11, 0.10, 0.11],
    [0.29, 0.91, 0.51, 0.20, 0.51],
]

train_texts = texts[:2]
train_labels = labels[:2]

eval_texts = texts[2:]
eval_labels = labels[2:]

tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

train_encodings = tokenizer(train_texts, padding="max_length", truncation=True, max_length=512)
eval_encodings = tokenizer(eval_texts, padding="max_length", truncation=True, max_length=512)

class TextClassifierDataset(Dataset):
    def __init__ (self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __len__ (self):
        return len(self.labels)

    def __getitem__ (self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item["labels"] = torch.tensor(self.labels[idx])
        return item

train_dataset = TextClassifierDataset(train_encodings, train_labels)
eval_dataset = TextClassifierDataset(eval_encodings, eval_labels)

model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", 
    problem_type="multi_label_classification",
    num_labels=5
)

training_arguments = TrainingArguments(
    output_dir=".",
    evaluation_strategy="epoch",
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    num_train_epochs=1,
)

trainer = Trainer(
    model=model,
    args=training_arguments,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

trainer.train()

```