# Training with class weights

**URL:** <https://discuss.huggingface.co/t/training-with-class-weights/62614>\
**Category:** 🤗Transformers\
**Created:** [November 17, 2023, 4:43pm UTC](https://discuss.huggingface.co/t/training-with-class-weights/62614 "2023-11-17T16:43:13Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ElisaOhOh](https://avatars.discourse-cdn.com/v4/letter/e/65b543/32.png) [@ElisaOhOh](https://discuss.huggingface.co/u/ElisaOhOh)\
**Post date:** [November 17, 2023, 4:43pm UTC](https://discuss.huggingface.co/t/training-with-class-weights/62614/1 "2023-11-17T16:43:13Z")

</div>

Hi everyone,

I am new to the huggingface library and I am fine-tuning a bert model for an imbalanced dataset with custom Trainer class and class weights:

```auto
from torch import nn
from transformers import Trainer

class CustomTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels")
        # forward pass
        outputs = model(**inputs)
        logits = outputs.get("logits")
        # compute custom loss (suppose one has 2 labels with different weights)
        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor([8.0, 1.0], device=model.device))
        loss = loss_fct(logits.view(-1, self.model.config.num_labels), labels.view(-1))
        return (loss, outputs) if return_outputs else loss

```

However, I got same results while training with class weights [10.0, 1.0] and [9.0, 1.0]. Is that normal? Or is there something wrong with my initialization? (I am initializing the model with model\_init option.)

Thanks for reading!

---

<div class="post-metadata">

**Author:** ![MattiLinnanvuori](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MattiLinnanvuori](https://discuss.huggingface.co/u/MattiLinnanvuori)\
**Post date:** [November 17, 2023, 4:51pm UTC](https://discuss.huggingface.co/t/training-with-class-weights/62614/2 "2023-11-17T16:51:34Z")

</div>

If your weights are similar to each other, I am not surprised that you get similar results.

---

<div class="post-metadata">

**Author:** ![ElisaOhOh](https://avatars.discourse-cdn.com/v4/letter/e/65b543/32.png) [@ElisaOhOh](https://discuss.huggingface.co/u/ElisaOhOh)\
**Post date:** [November 17, 2023, 5:15pm UTC](https://discuss.huggingface.co/t/training-with-class-weights/62614/3 "2023-11-17T17:15:03Z")

</div>

Thanks for your reply!

The problem is the results are exactly the same for class weights like 8.0, 9.0 and 10.0.

---

<div class="post-metadata">

**Author:** ![MattiLinnanvuori](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MattiLinnanvuori](https://discuss.huggingface.co/u/MattiLinnanvuori)\
**Post date:** [November 17, 2023, 5:32pm UTC](https://discuss.huggingface.co/t/training-with-class-weights/62614/4 "2023-11-17T17:32:08Z")

</div>

Do you mean Hugging Face pipeline for text classification classes are exactly the same or that scores are exactly the same?

---

<div class="post-metadata">

**Author:** ![ElisaOhOh](https://avatars.discourse-cdn.com/v4/letter/e/65b543/32.png) [@ElisaOhOh](https://discuss.huggingface.co/u/ElisaOhOh)\
**Post date:** [November 17, 2023, 5:50pm UTC](https://discuss.huggingface.co/t/training-with-class-weights/62614/5 "2023-11-17T17:50:29Z")

</div>

The resulting scores are exactly the same. I am working on a classification task so I looked at the classification report, and they are exactly the same for a various of class weights.

Maybe I did not initialize the model properly? I am using a jupyter notebook and I simply rerun the cells after changing the weights.

---

<div class="post-metadata">

**Author:** ![MattiLinnanvuori](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MattiLinnanvuori](https://discuss.huggingface.co/u/MattiLinnanvuori)\
**Post date:** [November 18, 2023, 10:09am UTC](https://discuss.huggingface.co/t/training-with-class-weights/62614/6 "2023-11-18T10:09:24Z")

</div>

I would check if the CustomTrainer class is really used.