# Class weights for bertForSequenceClassification

**URL:** <https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674>\
**Category:** Beginners\
**Created:** [October 21, 2020, 2:17pm UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674 "2020-10-21T14:17:29Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![jaecha](https://avatars.discourse-cdn.com/v4/letter/j/d9b06d/32.png) [@jaecha](https://discuss.huggingface.co/u/jaecha)\
**Post date:** [October 21, 2020, 2:17pm UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/1 "2020-10-21T14:17:29Z")

</div>

I have an unbalanced data with a couple of classes with relatively smaller sample sizes. I am wondering if there is a way to assign the class weights to BertFor SequenceClassification class, maybe in BertConfig ?, as we can do in nn.CrossEntropyLoss.

Thank you, in advance!

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 21, 2020, 2:34pm UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/2 "2020-10-21T14:34:12Z")

</div>

No, you need to compute the loss outside of the model for this. If you’re using `Trainer`, see [here](https://huggingface.co/transformers/main_classes/trainer.html) on how to change the loss form the default computed by the model.

---

<div class="post-metadata">

**Author:** ![jaecha](https://avatars.discourse-cdn.com/v4/letter/j/d9b06d/32.png) [@jaecha](https://discuss.huggingface.co/u/jaecha)\
**Post date:** [October 21, 2020, 3:00pm UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/3 "2020-10-21T15:00:31Z")

</div>

Hi Sylvain,

Glad to hear from you, outside of FastAI 🙂 Well, I am here as “Beginner” and will have to study more about Trainer. In the meantime, I tried the following:

1. run BertForSequenceClassification as usual
2. Take out logits from output (discard the loss from Bert run)
3. calculate new loss from nn.CrossEntropyLoss
4. and then calculate loss.backward()

Model runs okay, but I am not sure if this is a legitimate approach…

Thanks.

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\
**Post date:** [October 21, 2020, 3:03pm UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/4 "2020-10-21T15:03:25Z")

</div>

That is the correct approach!

---

<div class="post-metadata">

**Author:** ![jaecha](https://avatars.discourse-cdn.com/v4/letter/j/d9b06d/32.png) [@jaecha](https://discuss.huggingface.co/u/jaecha)\
**Post date:** [October 21, 2020, 3:22pm UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/5 "2020-10-21T15:22:06Z")

</div>

Fantastic!!! This means a lot that I got support from somebody like you, Sylvain! Thanks!! 😆

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [August 26, 2021, 3:25pm UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/6 "2021-08-26T15:25:08Z")

</div>

How do I take out the logits and calculate the new loss? Do you have a code example/snippet?

---

<div class="post-metadata">

**Author:** ![lucasresck](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lucasresck/32/3260_2.png) [@lucasresck](https://discuss.huggingface.co/u/lucasresck)\
**Post date:** [January 24, 2022, 4:02am UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/7 "2022-01-24T04:02:17Z")

</div>

Hi there,

I’m assuming you’re using the standard `BertForSequenceClassification`. So, instead of doing

```auto
outputs = model(**inputs)
loss = outputs['loss']

```

you do

```auto
outputs = model(**inputs)
logits = outputs['logits']
criterion = torch.nn.CrossEntropyLoss(weights=class_weights)
loss = criterion(logits, inputs['labels'])

```

assuming `inputs` is the dictionary that feeds the model. The loss function will leverage the class weights. Documentation for `CrossEntropyLoss` can be found [here](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html).

If, instead, you’re using `Trainer`, you’ll have to change the `compute_loss` method as informed previously:

> [@Class weights for bertForSequenceClassification](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/2):
>
> No, you need to compute the loss outside of the model for this. If you’re using Trainer, see [here](https://huggingface.co/transformers/main_classes/trainer.html) on how to change the loss form the default computed by the model.

As an example, modifying the [original implementation](https://github.com/huggingface/transformers/blob/6e058e84fd7a321a9a8edcc1dd36c6c0a3dd7ee7/src/transformers/trainer.py#L1962), it’d be something like

```auto
from transformers import Trainer
import torch

class MyTrainer(Trainer):
    def __init__ (self, class_weights, *args, **kwargs):
        super(). __init__ (*args, **kwargs)
        # You pass the class weights when instantiating the Trainer
        self.class_weights = class_weights

    def compute_loss(self, model, inputs, return_outputs=False):
        """
        How the loss is computed by Trainer. By default, all models return the loss in the first element.
        Subclass and override for custom behavior.
        """
        if self.label_smoother is not None and "labels" in inputs:
            labels = inputs.pop("labels")
        else:
            labels = None
        outputs = model(**inputs)
        # Save past state if it exists
        # TODO: this needs to be fixed and made cleaner later.
        if self.args.past_index >= 0:
            self._past = outputs[self.args.past_index]

        if labels is not None:
            loss = self.label_smoother(outputs, labels)
        else:
            # We don't use .loss here since the model may return tuples instead of ModelOutput.

            # Changes start here
            # loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]
            logits = outputs['logits']
            criterion = torch.nn.CrossEntropyLoss(weights=self.class_weights)
            loss = criterion(logits, inputs['labels'])
            # Changes end here

        return (loss, outputs) if return_outputs else loss

```

if you’re not using label smoothing.

---

<div class="post-metadata">

**Author:** ![xap](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xap/32/4537_2.png) [@xap](https://discuss.huggingface.co/u/xap)\
**Post date:** [May 24, 2022, 7:40pm UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/8 "2022-05-24T19:40:03Z")

</div>

@lucasresck as per your suggestion, I did the following

```auto
train_losses = []
num_mb_train = len(train_dataloader)

import torch.nn as nn
import numpy as np

total_preds=[]

for epoch in range(num_epochs):
  train_loss = 0

  
  for step, batch in enumerate(train_dataloader):
    
    model.train()
    
    batch = tuple(t.to(device) for t in batch)
    b_input_ids, b_input_mask,b_token_type, b_labels = batch

    outputs = model(b_input_ids, token_type_ids=b_token_type, attention_mask=b_input_mask, labels=b_labels)
    
    criterion = torch.nn.CrossEntropyLoss(weight=weights,reduction='mean')
    loss = criterion(outputs[1], b_labels)
    
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    scheduler.step()
        
    train_loss = loss.item()
    optimizer.zero_grad()
    
    train_losses.append(train_loss)

    if (step) % 50 == 0:
      print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}' 
             .format(epoch+1, num_epochs, step+1, total_steps, loss.item()))
    

```

Am I doing the right way or not?

---

<div class="post-metadata">

**Author:** ![afriedman412](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/afriedman412/32/4661_2.png) [@afriedman412](https://discuss.huggingface.co/u/afriedman412)\
**Post date:** [May 28, 2022, 10:31pm UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/9 "2022-05-28T22:31:41Z")

</div>

> [@lucasresck](#):
>
> `torch.nn.CrossEntropyLoss`

heads up, the param for `torch.nn.CrossEntropyLoss` is `weight` (not `weights`)

---

<div class="post-metadata">

**Author:** ![lucasresck](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lucasresck/32/3260_2.png) [@lucasresck](https://discuss.huggingface.co/u/lucasresck)\
**Post date:** [May 29, 2022, 12:42am UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/10 "2022-05-29T00:42:39Z")

</div>

Supposing `outputs[1]` refers to `outputs['logits']`, I believe it’s correct.

---

<div class="post-metadata">

**Author:** ![lucasresck](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lucasresck/32/3260_2.png) [@lucasresck](https://discuss.huggingface.co/u/lucasresck)\
**Post date:** [May 29, 2022, 12:45am UTC](https://discuss.huggingface.co/t/class-weights-for-bertforsequenceclassification/1674/11 "2022-05-29T00:45:47Z")

</div>

Thanks for the correction. I would edit the answer if possible.