# Training GPT-type models for classification tasks CausalLM vs SequenceClassification

**URL:** <https://discuss.huggingface.co/t/training-gpt-type-models-for-classification-tasks-causallm-vs-sequenceclassification/35982>\
**Category:** Models\
**Created:** [April 8, 2023, 7:03pm UTC](https://discuss.huggingface.co/t/training-gpt-type-models-for-classification-tasks-causallm-vs-sequenceclassification/35982 "2023-04-08T19:03:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![oqq09](https://avatars.discourse-cdn.com/v4/letter/o/f08c70/32.png) [@oqq09](https://discuss.huggingface.co/u/oqq09)\
**Post date:** [April 8, 2023, 7:03pm UTC](https://discuss.huggingface.co/t/training-gpt-type-models-for-classification-tasks-causallm-vs-sequenceclassification/35982/1 "2023-04-08T19:03:57Z")

</div>

I see examples like:

[Fine-tuning GPT-J for text entailment](https://console.paperspace.com/github/gradient-ai/Graphcore-Pytorch?machine=IPU-POD16&container=graphcore%2Fpytorch-jupyter%3A3.1.0-ubuntu-20.04-20230104&file=%2Fgptj-text-generation%2Ffinetuning.ipynb&utm_source=Graphcore+Blog&utm_medium=Blog+Walkthrough&utm_campaign=GPT-J&utm_term=GPT-J+finetuning&utm_content=text-entailment-GPT-J-finetuning)

Where CausalLM class is used to fine tune the model on the task instead of the SequenceClassification. What is the purpose of doing that?

---

<div class="post-metadata">

**Author:** ![oqq09](https://avatars.discourse-cdn.com/v4/letter/o/f08c70/32.png) [@oqq09](https://discuss.huggingface.co/u/oqq09)\
**Post date:** [April 10, 2023, 10:31pm UTC](https://discuss.huggingface.co/t/training-gpt-type-models-for-classification-tasks-causallm-vs-sequenceclassification/35982/2 "2023-04-10T22:31:43Z")

</div>

I would like to add more context to my question and maybe narrow it down to where I am facing a problem fine tuning a similar model (distilGPT2) using a custom dataset.

I am trying to figure out how to fine-tune a GPT-type model using the CausalLM class;  
First, the `"input_ids"` are the encoded form of text in the format below:

```auto
f"mnli hypothesis: {hypothesis} premise: {premise} target: {class_label} <|endoftext|>"

```

Second, the training set is grouped in a similar manner to the `group_text` function used in the huggingface notebook example for fine-tuning CausalLM models ([[Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/language_modeling.ipynb)]([https://How](https://How) to fine-tune a model on language modeling | [huggingface.co](http://huggingface.co))) which also creates the `"labels"` by shifting the input\_id by one token.

How should I pre-process the eval set? Do I use the same method as I did on the training set? The concept of grouping the text using the `group_text` function is making it hard for me to understand how will the model know that I want it to focus on generating the `"class_label"` when it is getting fine-tuned!

---

<div class="post-metadata">

**Author:** ![daminho](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/daminho/32/29506_2.png) [@daminho](https://discuss.huggingface.co/u/daminho)\
**Post date:** [August 7, 2024, 9:17am UTC](https://discuss.huggingface.co/t/training-gpt-type-models-for-classification-tasks-causallm-vs-sequenceclassification/35982/3 "2024-08-07T09:17:28Z")

</div>

Hi, I’m currently having the same question with you. Could you share your solution?