# Which loss function in bertforsequenceclassification regression **URL:** \ **Category:** Beginners\ **Created:** [October 8, 2020, 8:22pm UTC](https://discuss.huggingface.co/t/which-loss-function-in-bertforsequenceclassification-regression/1432 "2020-10-08T20:22:15Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [October 8, 2020, 8:22pm UTC](https://discuss.huggingface.co/t/which-loss-function-in-bertforsequenceclassification-regression/1432/1 "2020-10-08T20:22:15Z")
BertForSequenceClassification can be used for regression when number of classes is set to 1. The documentation says that BertForSequenceClassification calculates cross-entropy loss for classification. What kind of loss does it return for regression? (I’ve been assuming it is root mean square error, but I read recently that there are several other possibilities such as Huber or Negative Log Likelihood.) Which is it? How should I find out / where is the code? ---
**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\ **Post date:** [October 9, 2020, 8:28am UTC](https://discuss.huggingface.co/t/which-loss-function-in-bertforsequenceclassification-regression/1432/2 "2020-10-09T08:28:56Z")
This is the [GitHub](https://github.com/huggingface/transformers/blob/9aeacb58bab321bc21c24bbdf7a24efdccb1d426/src/transformers/modeling_bert.py) link At line 1354, you have the condition to check the labels (if it is one or more) **if self.num\_labels == 1** : # We are doing regression loss\_fct = MSELoss() loss = loss\_fct(logits.view(-1), labels.view(-1)) **else** : loss\_fct = CrossEntropyLoss() loss = loss\_fct(logits.view(-1, self.num\_labels), labels.view(-1)) ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [October 9, 2020, 9:12am UTC](https://discuss.huggingface.co/t/which-loss-function-in-bertforsequenceclassification-regression/1432/3 "2020-10-09T09:12:27Z")
You can select the lines that you are interested in on a Github code page, then click on the three dots and select copy permalink: > ---
**Author:** ![Karthik12](https://avatars.discourse-cdn.com/v4/letter/k/dec6dc/32.png) [@Karthik12](https://discuss.huggingface.co/u/Karthik12)\ **Post date:** [October 9, 2020, 9:50am UTC](https://discuss.huggingface.co/t/which-loss-function-in-bertforsequenceclassification-regression/1432/4 "2020-10-09T09:50:54Z")
Thanks @BramVanroy, this makes the code easier to read. I will do so, in future. ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [October 9, 2020, 1:06pm UTC](https://discuss.huggingface.co/t/which-loss-function-in-bertforsequenceclassification-regression/1432/5 "2020-10-09T13:06:51Z")
Thank you ! ---
**Author:** ![theudster](https://avatars.discourse-cdn.com/v4/letter/t/97f17d/32.png) [@theudster](https://discuss.huggingface.co/u/theudster)\ **Post date:** [February 24, 2021, 10:49am UTC](https://discuss.huggingface.co/t/which-loss-function-in-bertforsequenceclassification-regression/1432/6 "2021-02-24T10:49:36Z")
Can you add parameters to the loss function through transformer? for example, add weights to each of the classes? ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [February 24, 2021, 7:25pm UTC](https://discuss.huggingface.co/t/which-loss-function-in-bertforsequenceclassification-regression/1432/7 "2021-02-24T19:25:27Z")
Hi theudster, the pytorch docs for CrossEntropyLoss suggest that you can add a weight tensor [CrossEntropyLoss — PyTorch 1.7.1 documentation](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) . What happens if you try it? ---
**Author:** ![theudster](https://avatars.discourse-cdn.com/v4/letter/t/97f17d/32.png) [@theudster](https://discuss.huggingface.co/u/theudster)\ **Post date:** [February 25, 2021, 1:25pm UTC](https://discuss.huggingface.co/t/which-loss-function-in-bertforsequenceclassification-regression/1432/8 "2021-02-25T13:25:31Z")
I haven’t tried that because I am trying to implement everything through the Trainer method