# Multilabel classification using LLMs **URL:** \ **Category:** Beginners\ **Created:** [April 1, 2024, 2:13pm UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671 "2024-04-01T14:13:25Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Utkarsh-Tiwari](https://avatars.discourse-cdn.com/v4/letter/u/958977/32.png) [@Utkarsh-Tiwari](https://discuss.huggingface.co/u/Utkarsh-Tiwari)\ **Post date:** [April 1, 2024, 2:13pm UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/1 "2024-04-01T14:13:25Z")
I needed to know what’s the best way to finetune LLM models for multiclass classification tasks where there are more than 100 classes. I assume that ‘Text Generation’ is the main functionality of these LLMs and most of the coding examples and documentations show the ‘Text Generation’ as the example only. I know that I can generate those labels by finetuning these ‘Text Generation’ models on my dataset, but this will only train the LLMs on the labels that are present in the train dataset and still there are labels which are missing from my train dataset. The ideal scenario would be to have datasets for all the classes in my train dataset but it’s not the case as of now. Also, there might be instances where these text generation models would generate new classes of their own which doesn’t exist at all. I have seen on HF that there are model classes of these LLMs like ‘LlamaforSequenceClassification’ with sequence classification heads but haven’t found any example of their implementation. I know that BERT models are used for the Sequence Classification tasks but can I do it using any of the LLMs? If possible, please share any example for the same. ---
**Author:** ![MattiLinnanvuori](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MattiLinnanvuori](https://discuss.huggingface.co/u/MattiLinnanvuori)\ **Post date:** [April 8, 2024, 2:50pm UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/2 "2024-04-08T14:50:39Z")
Below I attach an example of multilabel classification with an LLM model. [https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine\_tuning\_BERT\_(and\_friends)\_for\_multi\_label\_text\_classification.ipynb](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb) ---
**Author:** ![Utkarsh-Tiwari](https://avatars.discourse-cdn.com/v4/letter/u/958977/32.png) [@Utkarsh-Tiwari](https://discuss.huggingface.co/u/Utkarsh-Tiwari)\ **Post date:** [April 9, 2024, 7:52am UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/3 "2024-04-09T07:52:55Z")
Thank you for your reply. I wanted to know if I can use the LLMs like Llama2, Mistal, Phi2 (Decoder only models) for text classification just like you did here with BERT (which is an encoder model). In the Llama2 documentation on HF, they have mentioned a class - [LlamaForSequenceClassification](https://huggingface.co/docs/transformers/main/en/model_doc/llama2#transformers.LlamaForSequenceClassification) But I am not sure how to utilize this class for sequence classification purpose. ---
**Author:** ![swtb](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@swtb](https://discuss.huggingface.co/u/swtb)\ **Post date:** [April 9, 2024, 9:05am UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/4 "2024-04-09T09:05:17Z")
I have used Mistral for NER which is a word level classification task. I used RAG to retrieve similar queries (and their labels) and show the decoder model what I expect the output to be. You can then pass the query and it should classify it according to the examples you have shown it. I will say while this works…it is certainly more native to do this with and encoder like BERT. ---
**Author:** ![juhoinkinen](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/juhoinkinen/32/32436_2.png) [@juhoinkinen](https://discuss.huggingface.co/u/juhoinkinen)\ **Post date:** [April 9, 2024, 9:24am UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/5 "2024-04-09T09:24:32Z")
If non-LLM approaches suit you, you could consider [Annif](https://github.com/NatLibFi/Annif), see the demo at [annif.org](https://annif.org/). Annif is developed for extreme multilabel classification of texts, so it is most suitable if there are thousands or tens of thousands labels to choose from. > but this will only train the LLMs on the labels that are present in the train dataset and still there are labels which are missing from my train dataset. Annif utilizes two kinds of algorithms in its backends: associative and lexical. An associative algorithm suggests only the labels it has seen on its training set, whereas a lexical algorithm can suggest any label from the vocabulary, because it learns features on based on word positions etc. See [Backend: MLLM · NatLibFi/Annif Wiki · GitHub](https://github.com/NatLibFi/Annif/wiki/Backend%3A-MLLM) for more details. (That said, Annif could in future use some LLM or BERT for well-performing zero-shot analysis, which is why I’m interested in these topics.) ---
**Author:** ![MattiLinnanvuori](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MattiLinnanvuori](https://discuss.huggingface.co/u/MattiLinnanvuori)\ **Post date:** [April 9, 2024, 12:14pm UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/6 "2024-04-09T12:14:52Z")
You can utilise a decoder-only model like LlamaForSequenceClassification like in the link above. If HuggingFace has created a sequence classification class, you can use it like above. ---
**Author:** ![Utkarsh-Tiwari](https://avatars.discourse-cdn.com/v4/letter/u/958977/32.png) [@Utkarsh-Tiwari](https://discuss.huggingface.co/u/Utkarsh-Tiwari)\ **Post date:** [April 11, 2024, 9:48am UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/7 "2024-04-11T09:48:40Z")
Thank you, I will try Annif and see if this helps for my use case. ---
**Author:** ![Utkarsh-Tiwari](https://avatars.discourse-cdn.com/v4/letter/u/958977/32.png) [@Utkarsh-Tiwari](https://discuss.huggingface.co/u/Utkarsh-Tiwari)\ **Post date:** [April 11, 2024, 9:52am UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/8 "2024-04-11T09:52:36Z")
I guess that using RAG for text classification would not generate most accurate predictions as compared to fine tuning the LLM on my dataset. ---
**Author:** ![Utkarsh-Tiwari](https://avatars.discourse-cdn.com/v4/letter/u/958977/32.png) [@Utkarsh-Tiwari](https://discuss.huggingface.co/u/Utkarsh-Tiwari)\ **Post date:** [April 11, 2024, 9:53am UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/9 "2024-04-11T09:53:56Z")
I will figure this out and will post here if it works or not. ---
**Author:** ![Utkarsh-Tiwari](https://avatars.discourse-cdn.com/v4/letter/u/958977/32.png) [@Utkarsh-Tiwari](https://discuss.huggingface.co/u/Utkarsh-Tiwari)\ **Post date:** [June 6, 2024, 11:30am UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/10 "2024-06-06T11:30:57Z")
I tried using the Llama for Sequence Classification head, but it turns out that the predictions were not upto the mark and worse than the BERT classification model predictions. It’s better to use the AutoModelForCausalLM class for finetuning the decoder only LLMs as they generate better predictions than BERT and LLM model with sequence classification heads. ---
**Author:** ![VidaRoha](https://avatars.discourse-cdn.com/v4/letter/v/838e76/32.png) [@VidaRoha](https://discuss.huggingface.co/u/VidaRoha)\ **Post date:** [June 6, 2024, 10:58pm UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/11 "2024-06-06T22:58:26Z")
Hi, I am now having very similar task. I am not familiar with AutoModelForCausalLM but I hope you succeed in what you are doing. If it’s okay, please let me know if it works out. I am trying to use RAG+ GPT approach. Whether it works out or not, I’ll post the comment here. ---
**Author:** ![Utkarsh-Tiwari](https://avatars.discourse-cdn.com/v4/letter/u/958977/32.png) [@Utkarsh-Tiwari](https://discuss.huggingface.co/u/Utkarsh-Tiwari)\ **Post date:** [June 7, 2024, 7:01am UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/12 "2024-06-07T07:01:58Z")
Refer this: [Google Colab](https://colab.research.google.com/drive/1ggaa2oRFphdBmqIjSEbnb_HGkcIRC2ZB?usp=sharing) This notebook is available in the Hugging Face documentation of Llama2 model ( [Llama2 (huggingface.co)](https://huggingface.co/docs/transformers/main/model_doc/llama2)), and I think this is the most appropriate way for text classification if you want to use Decoder only models for text classification tasks. RAG and AutoModelForSequenceClassification head on top of these decoder only LLMs, won’t give the best results for text classification. ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [June 7, 2024, 7:02pm UTC](https://discuss.huggingface.co/t/multilabel-classification-using-llms/79671/13 "2024-06-07T19:02:48Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.