# LLaMa3.1 8B Instruct Prompt Tuning for Text Classification doesn't improve test accuracy

**URL:** <https://discuss.huggingface.co/t/llama3-1-8b-instruct-prompt-tuning-for-text-classification-doesnt-improve-test-accuracy/108838>\
**Category:** Models\
**Created:** [September 26, 2024, 5:33pm UTC](https://discuss.huggingface.co/t/llama3-1-8b-instruct-prompt-tuning-for-text-classification-doesnt-improve-test-accuracy/108838 "2024-09-26T17:33:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rbelanec](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rbelanec/32/32117_2.png) [@rbelanec](https://discuss.huggingface.co/u/rbelanec)\
**Post date:** [September 26, 2024, 5:33pm UTC](https://discuss.huggingface.co/t/llama3-1-8b-instruct-prompt-tuning-for-text-classification-doesnt-improve-test-accuracy/108838/1 "2024-09-26T17:33:53Z")

</div>

Hello everyone!

I am trying to fine-tune **LLaMa3.1 8B Instruct for text classification with prompt tuning** from peft library (prepending a trainable matrix before input embeddings). For instance, I am using the QNLI dataset to classify whether a question and sentence are in entailment.

I have preprocessed the dataset so that each data looks like the following:

```auto
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 26 Jul 2024

<|eot_id|><|start_header_id|>user<|end_header_id|>

Classify the question and sentence pair into labels: entailment, not entailment. Reply only the corresponding label.
question: What is the name of a former Asian Portuguese colony?
sentence: The country has a tiny Chinese population.
label:<|eot_id|><|start_header_id|>assistant<|end_header_id|>

not entailment<|eot_id|>

```

In test data, I remove the label and leave only the generation prompt.

When I evaluate the zero-shot performance of LLaMa3.1 with the text-generation pipeline I get around **74% accuracy** (which is already really good).

For training, I am using SFTTrainer and SFTConfig. You can find my full config [here](https://github.com/Wicwik/task-prompt-vectors/blob/main/configs/prompt_tuning/single-task/llama31_per_task/llama31_8b_qnli.toml) (there are also some custom parameters, but mostly nothing out of ordinary).

The valid and train loss go down to less than 1 after one epoch. After one epoch I evaluated the test set again and the **accuracy was still around 74%.**

My question is does the LLaMa3 model ignore the soft-prompt? Or was it just undertrained and I should keep training it for more epochs? Since the loss is decreasing I suppose that it is not ignoring it during training.

Is it even possible to use prompt tuning with instruction fine-tuned models? I also tried manual hyperparameter tunning (mostly learning rate and soft-prompt length) but the results were mostly the same.

If anybody has more experience with prompt tuning and instruction-tuned autoregressive models I would be thankful if they could point me in the right direction.

Thank you!

edits: added info about pipelines that I thought may be useful

---

<div class="post-metadata">

**Author:** ![rbelanec](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rbelanec/32/32117_2.png) [@rbelanec](https://discuss.huggingface.co/u/rbelanec)\
**Post date:** [September 30, 2024, 12:32pm UTC](https://discuss.huggingface.co/t/llama3-1-8b-instruct-prompt-tuning-for-text-classification-doesnt-improve-test-accuracy/108838/2 "2024-09-30T12:32:02Z")

</div>

🎆 So I was able to fix this issue! 🎆

I have been using the **peft library of version 0.12.0.** In this version the problem was in the implementation of prepare\_inputs\_for\_generation of the PeftModelForCausalLM class. As you can see from [this line](https://github.com/huggingface/peft/blob/v0.12.0/src/peft/peft_model.py#L1698), the soft-prompt is only added when “past\_key\_values” are None. I fixed this by using use\_cache=False and a quick if statement:

```auto
if model_kwargs["use_cache"] == False:
            model_kwargs["past_key_values"] = None

```

In version 0.13.0 it has been already fixed by adding new flag require\_prompt\_injection that is defined in [this line](https://github.com/huggingface/peft/blob/v0.13.0/src/peft/peft_model.py#L1769). This allows to use soft-prompt-based methods also with 🤗 caches. I haven’t tested it yet tho. Hope this helps someone, It took me a day or so to find out 😃

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 30, 2024, 12:54pm UTC](https://discuss.huggingface.co/t/llama3-1-8b-instruct-prompt-tuning-for-text-classification-doesnt-improve-test-accuracy/108838/3 "2024-09-30T12:54:15Z")

</div>

There are a lot of people using that version of the PEFT library in Spaces. And with LLM.  
The good news is that there is a fixed version.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [October 1, 2024, 12:54am UTC](https://discuss.huggingface.co/t/llama3-1-8b-instruct-prompt-tuning-for-text-classification-doesnt-improve-test-accuracy/108838/4 "2024-10-01T00:54:25Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.