# XLM-Roberta for many-topic classification

**URL:** <https://discuss.huggingface.co/t/xlm-roberta-for-many-topic-classification/12638>\
**Category:** Beginners\
**Created:** [December 8, 2021, 2:13pm UTC](https://discuss.huggingface.co/t/xlm-roberta-for-many-topic-classification/12638 "2021-12-08T14:13:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![shomy303](https://avatars.discourse-cdn.com/v4/letter/s/858c86/32.png) [@shomy303](https://discuss.huggingface.co/u/shomy303)\
**Post date:** [December 8, 2021, 2:13pm UTC](https://discuss.huggingface.co/t/xlm-roberta-for-many-topic-classification/12638/1 "2021-12-08T14:13:40Z")

</div>

Hi,  
I am working on a multi-label topic classifier that can classify webpages into some of our ~100 topics. The classifier currently uses a basic Neural Network and I wish to adapt the XLM-R model provided by Huggingface to give the classifier multi-lingual capabilities.

However, when I train a classifier using XLM-R the performance (using pr\_auc) is worse than that of the classifier using the basic Neural Network.

What can I do to improve the performance of a transformer-based Neural Network model?

---

<div class="post-metadata">

**Author:** ![carschno](https://avatars.discourse-cdn.com/v4/letter/c/c57346/32.png) [@carschno](https://discuss.huggingface.co/u/carschno)\
**Post date:** [December 31, 2021, 12:51pm UTC](https://discuss.huggingface.co/t/xlm-roberta-for-many-topic-classification/12638/2 "2021-12-31T12:51:45Z")

</div>

I don’t have an answer, unfortunately, but I have the same issue and can add some details. I have a classification problem with 60 classes. I have training data of ~70k documents, unfortunately, with very unbalanced class distribution. My baseline is a FastText classifier trained on the same data which achieves an accuracy of ~0.45. The majority of the documents is in English, but some are in other languages (all

I have followed the [tutorial for fine-tuning pre-trained classification models](https://huggingface.co/docs/transformers/training) using the Trainer API. I have not changed much from the example other than the number of labels, and the model.

Running on a Colab notebook with GPU, training time for a single epoch is roughly 4h. I’ve run intermediate evaluations every 500 steps, and the accuracy is around 0.04, no changes (neither positive nor negative). Same happens when using a different model (e.g. `bert-base-cased`, as shown in the tutorial).

My suspicion is that the model does not learn anything at all, the accuracy is very close to random choice.  
What am I missing? Is there a more suitable, up-to-date tutorial somewhere?