# Xlm-roberta-base predicting always same class, other models don't

**URL:** <https://discuss.huggingface.co/t/xlm-roberta-base-predicting-always-same-class-other-models-dont/26647>\
**Category:** Intermediate\
**Created:** [November 24, 2022, 12:17pm UTC](https://discuss.huggingface.co/t/xlm-roberta-base-predicting-always-same-class-other-models-dont/26647 "2022-11-24T12:17:15Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![lucadini](https://avatars.discourse-cdn.com/v4/letter/l/d9b06d/32.png) [@lucadini](https://discuss.huggingface.co/u/lucadini)\
**Post date:** [November 24, 2022, 12:17pm UTC](https://discuss.huggingface.co/t/xlm-roberta-base-predicting-always-same-class-other-models-dont/26647/1 "2022-11-24T12:17:15Z")

</div>

Hi all,  
I’m currently trying to fine-tune xlm-roberta-base for a binary classification task, using a pretty standard code:

```auto
    [...]
    data_files = {'train': train_path, 'test': test_path}
    model_name = 'xlm-roberta-base'
    tokenizer = AutoTokenizer.from_pretrained(model_name)

    train_dataset = create_dataset(data_files['train'], tokenizer, shuffle=True) #just load dataset from file and tokenizes text
    test_dataset = create_dataset(data_files['test'], tokenizer)

    model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)
    metric = evaluate.load("accuracy")

    def compute_metrics(eval_pred):
        logits, labels = eval_pred
        predictions = np.argmax(logits, axis=-1)
        return metric.compute(predictions=predictions, references=labels)

    training_args = TrainingArguments(output_dir=output_dir, num_train_epochs=5, 
     per_device_train_batch_size=16, per_device_eval_batch_size=16, data_seed=42, 
     logging_dir='logs', logging_strategy='epoch', save_strategy='no', 
     evaluation_strategy='epoch')

    trainer = Trainer(model=model, args=training_args, train_dataset=train_dataset, 
     compute_metrics=compute_metrics, eval_dataset=test_dataset)

    trainer.train()
    trainer.save_model()
    [...]

```

I’m using a balanced dataset with 50% examples of class 0 and 50% of class 1. On the evaluation set, the model always predict one class. If I change model to another, for example “bert-base-cased” it reaches an accuracy of 92%. What am I missing?

Thanks in advace!

---

<div class="post-metadata">

**Author:** ![qyubee](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/qyubee/32/16583_2.png) [@qyubee](https://discuss.huggingface.co/u/qyubee)\
**Post date:** [June 5, 2023, 7:07pm UTC](https://discuss.huggingface.co/t/xlm-roberta-base-predicting-always-same-class-other-models-dont/26647/2 "2023-06-05T19:07:01Z")

</div>

Hello,  
did you find the solution ?

---

<div class="post-metadata">

**Author:** ![lucadini](https://avatars.discourse-cdn.com/v4/letter/l/d9b06d/32.png) [@lucadini](https://discuss.huggingface.co/u/lucadini)\
**Post date:** [June 7, 2023, 8:45am UTC](https://discuss.huggingface.co/t/xlm-roberta-base-predicting-always-same-class-other-models-dont/26647/3 "2023-06-07T08:45:46Z")

</div>

I lowered the learning rate by one order of magnitude and it worked!