# Data format for BertForSequenceClassification with num\_labels \> 2

**URL:** <https://discuss.huggingface.co/t/data-format-for-bertforsequenceclassification-with-num-labels-2/4156>\
**Category:** Beginners\
**Created:** [March 4, 2021, 5:03pm UTC](https://discuss.huggingface.co/t/data-format-for-bertforsequenceclassification-with-num-labels-2/4156 "2021-03-04T17:03:10Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![maxpower](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@maxpower](https://discuss.huggingface.co/u/maxpower)\
**Post date:** [March 4, 2021, 5:03pm UTC](https://discuss.huggingface.co/t/data-format-for-bertforsequenceclassification-with-num-labels-2/4156/1 "2021-03-04T17:03:10Z")

</div>

Hi,  
I have a multilabel task (num\_labels=8) and I want to use BertForSequenceClassification using Trainer to train the model.

But I get the following error:

> ValueError: Expected input batch\_size (8) to match target batch\_size (64).

I assume that the problem is the data format of the labels. Currently, my label is a 8-dim list (e.g., [1,0,0,0,0,1,0,0]).

What is the right format for the label data?

Here my code:

> class EmotionDataset(torch.utils.data.Dataset):  
> def **init** (self, encodings, labels):  
> self.encodings = encodings  
> self.labels = labels
> 
> ```
> def __getitem__ (self, idx):
> item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
> item['labels'] = torch.tensor(self.labels[idx])
> return item
> 
> def __len__ (self):
> return len(self.labels)
> 
> ```
> 
> MODEL\_NAME = ‘dbmdz/bert-base-german-uncased’
> 
> tokenizer = AutoTokenizer.from\_pretrained(MODEL\_NAME)  
> model = BertForSequenceClassification.from\_pretrained(MODEL\_NAME, num\_labels=8)
> 
> ### tokenize data
> 
> dataset\_train = Dataset.from\_pandas(df\_train)  
> train\_encodings = tokenizer(dataset\_train['text], truncation=True, padding=True)  
> train\_dataset = EmotionDataset(train\_encodings, dataset\_train['label])
> 
> training\_args = TrainingArguments(  
> output\_dir=‘./results’, # output directory  
> num\_train\_epochs=1, # total # of training epochs  
> per\_device\_train\_batch\_size=8, # batch size per device during training  
> per\_device\_eval\_batch\_size=32, # batch size for evaluation  
> warmup\_steps=500, # number of warmup steps for learning rate scheduler  
> weight\_decay=0.01, # strength of weight decay  
> logging\_dir=‘./logs’, # directory for storing logs  
> )
> 
> trainer = Trainer(  
> model=model, # the instantiated 🤗 Transformers model to be trained  
> args=training\_args, # training arguments, defined above  
> train\_dataset=train\_dataset, # training dataset  
> eval\_dataset=test\_dataset # evaluation dataset  
> )
> 
> \_ = trainer.train()  
> trainer.evaluate()

Thanks,  
Max

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [March 4, 2021, 5:29pm UTC](https://discuss.huggingface.co/t/data-format-for-bertforsequenceclassification-with-num-labels-2/4156/2 "2021-03-04T17:29:12Z")

</div>

Hi @maxpower, I think the format of your dataset is fine but I think you have to change the model’s loss function to use a sigmoid instead of a softmax on the logits (i.e. `BCEWithLogitsLoss`). You can see a skeleton + hacky Colab in this thread: [Fine-Tune for MultiClass or MultiLabel-MultiClass - #8 by lewtun](https://discuss.huggingface.co/t/fine-tune-for-multiclass-or-multilabel-multiclass/4035/8)

---

<div class="post-metadata">

**Author:** ![maxpower](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@maxpower](https://discuss.huggingface.co/u/maxpower)\
**Post date:** [March 5, 2021, 9:46am UTC](https://discuss.huggingface.co/t/data-format-for-bertforsequenceclassification-with-num-labels-2/4156/3 "2021-03-05T09:46:16Z")

</div>

Perfect, it works. Thanks so much!

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [March 5, 2021, 10:13am UTC](https://discuss.huggingface.co/t/data-format-for-bertforsequenceclassification-with-num-labels-2/4156/4 "2021-03-05T10:13:01Z")

</div>

FYI I just posted a more elegant solution in the thread that just subclasses `Trainer` and overrides the `compute_loss` function (you can see it in action in the Colab notebook too 🙂)

---

<div class="post-metadata">

**Author:** ![opey](https://avatars.discourse-cdn.com/v4/letter/o/2acd7d/32.png) [@opey](https://discuss.huggingface.co/u/opey)\
**Post date:** [July 21, 2021, 5:01pm UTC](https://discuss.huggingface.co/t/data-format-for-bertforsequenceclassification-with-num-labels-2/4156/5 "2021-07-21T17:01:48Z")

</div>

Hi lewtun,  
Thanks for your help so far.  
But I’m having issues getting it to work for multiclass classification.  
The custommetric in the notebook only works for multilabel classification.  
Is there anything I need to do please?

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\
**Post date:** [August 2, 2021, 6:14pm UTC](https://discuss.huggingface.co/t/data-format-for-bertforsequenceclassification-with-num-labels-2/4156/6 "2021-08-02T18:14:35Z")

</div>

hey @opey for ordinary multiclass classification you can follow the official tutorial [here](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb) or just run one of the scripts in the examples [here](https://github.com/huggingface/transformers/tree/master/examples/pytorch/text-classification)

hope that helps!