# Different results from checkpoint evaluation when loading fine-tuned LLM model

**URL:** <https://discuss.huggingface.co/t/different-results-from-checkpoint-evaluation-when-loading-fine-tuned-llm-model/43106>\
**Category:** Intermediate\
**Created:** [June 13, 2023, 6:02pm UTC](https://discuss.huggingface.co/t/different-results-from-checkpoint-evaluation-when-loading-fine-tuned-llm-model/43106 "2023-06-13T18:02:18Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ilektram](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ilektram](https://discuss.huggingface.co/u/ilektram)\
**Post date:** [June 13, 2023, 6:02pm UTC](https://discuss.huggingface.co/t/different-results-from-checkpoint-evaluation-when-loading-fine-tuned-llm-model/43106/1 "2023-06-13T18:02:19Z")

</div>

I am finetuning Llama for binary sequence classification with PEFT & Lora using the Trainer class. Loss seems to decrease nicely and accuracy on the validation data reaches ~90% in the final epoch: ` { "epoch": 4.98, "eval_accuracy": 0.9346576058546785, "eval_loss": 0.18449442088603973, "eval_runtime": 496.2064, "eval_samples_per_second": 7.711, "eval_steps_per_second": 0.965, "step": 595 }`  
After the training is over I save the model as follows:`

```auto
model.state_dict = (
    lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())
). __get__ (model, type(model))

model = torch.compile(model)

trainer.train()
model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)
torch.save(model.state_dict(), "torch_openllama_saver")

```

I subsequently try to reload the model and reproduce the evaluation result on the same validation set. However, I get an **accuracy of 55%** this time. I load the model as follows:

```auto
CUTOFF_LEN = 512

if torch.cuda.is_available():
    device = "cuda"
else:
    device = "cpu"

try:
    if torch.backends.mps.is_available():
        device = "mps"
except: # noqa: E722
    pass

import sys
import textwrap

import torch
from peft import PeftModel, PeftConfig
from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig
from transformers.generation.utils import GreedySearchDecoderOnlyOutput

if torch.cuda.is_available():
    device = "cuda"
else:
    device = "cpu"

try:
    if torch.backends.mps.is_available():
        device = "mps"
except: # noqa: E722
    pass

load_8bit = True
# Load peft config for pre-trained checkpoint etc.
peft_model_id = "results/experiments_openllama"
config = PeftConfig.from_pretrained(peft_model_id)
config.inference_mode = True

base_model = config.base_model_name_or_path
lora_weights = peft_model_id
if device == "cuda":
    model = LlamaForSequenceClassification.from_pretrained(
        base_model,
        load_in_8bit=load_8bit,
        torch_dtype=torch.float16,
        device_map="auto",
    )
    model = PeftModel.from_pretrained(
        model, lora_weights, torch_dtype=torch.float16, config=config
    )
elif device == "mps":
    model = LlamaForSequenceClassification.from_pretrained(
        base_model, device_map={"": device}, torch_dtype=torch.float16,
    )
    model = PeftModel.from_pretrained(
        model,
        lora_weights,
        device_map={"": device},
        torch_dtype=torch.float16,
        config=config,
    ) # must set inference_mode=True
else:
    model = LlamaForSequenceClassification.from_pretrained(
        base_model, device_map={"": device}, low_cpu_mem_usage=True
    )
    model = PeftModel.from_pretrained(
        model, lora_weights, device_map={"": device}, config=config
    )

model.load_state_dict(
    torch.load(os.path.join(lora_weights, "adapter_model.bin")), strict=False
)

tokenizer = LlamaTokenizer.from_pretrained(peft_model_id)

model.config.pad_token_id = tokenizer.pad_token_id
model.config.bos_token_id = tokenizer.bos_token_id
model.config.eos_token_id = tokenizer.eos_token_id
model.config.sep_token_id = tokenizer.sep_token_id
model.config.unk_token_id = tokenizer.pad_token_id

```

Why is that & what am I doing wrong? Is the original evaluation accuracy incorrect? Or is there an issue with loading the trained model weights?

---

<div class="post-metadata">

**Author:** ![00BER](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/00ber/32/16876_2.png) [@00BER](https://discuss.huggingface.co/u/00BER)\
**Post date:** [June 15, 2023, 6:47pm UTC](https://discuss.huggingface.co/t/different-results-from-checkpoint-evaluation-when-loading-fine-tuned-llm-model/43106/2 "2023-06-15T18:47:03Z")

</div>

I recently ran into the problem where the LORA adapters weren’t being saved correctly.  
If you are running finetuning code similar to the alpaca-lora repository, apparently there is a bug where the adapter weights aren’t saved.

Check out these issues:

> <https://github.com/tloen/alpaca-lora/issues/477>
>
> In the finetune.py, lin line 246, where the original \`model.state\_dict\` is chan…ged into a lambda function of \`get\_peft\_model\_state\_dict()\`
> 
> \`get\_peft\_model\_state\_dict()\` will retuen a dict of only the Lora parameter and the keys of the parameter is regularized, the \`adapter\_name\` in the key string is removed. 
> \[\`to\_return = {k.replace(f".{adapter\_name}", ""): v for k, v in to\_return.items()}\`\](https://github.com/huggingface/peft/blob/3714aa2fff158fdfa637b2b65952580801d890b2/src/peft/utils/save\_and\_load.py#LL76C1-L76C1)
> 
> Then when saving the model after Training, In the finetune.py line 275, use the \`model.save\_pretrained()\` function, which is implemented by the \`PeftModel.save\_pretrained()\`. Inside the \`PeftModel.save\_pretrained()\`, \` get\_peft\_model\_state\_dict()\` is used again. 
> 
> However, because the \`model.state\_dict\` is already changed! The dict returened by it is regularized (changed keys' name), the the \` get\_peft\_model\_state\_dict()\` inside the \`PeftModel.save\_pretrained()\` will remove all the parameter whose keys string doesn't contain the \`adapter\_name\`, which is all! It will result in the final state\_dict to be save is None!!!
> \[\`to\_return = {k: v for k, v in to\_return.items() if (("lora\_" in k and adapter\_name in k) or ("bias" in k))}\`\](https://github.com/huggingface/peft/blob/3714aa2fff158fdfa637b2b65952580801d890b2/src/peft/utils/save\_and\_load.py#LL52C9-L52C116)

> <https://github.com/huggingface/peft/issues/286>
>
> I recently found that when fine-tuning using \[alpaca-lora\](https://github.com/tl…oen/alpaca-lora), \[\`model.save\_pretrained()\`\](https://github.com/tloen/alpaca-lora/blob/8d58d37b65501eb07a1397a10ca5e80834b626f1/finetune.py#L268) will save a \`adapter\_model.bin\` that is only 443 B.
> 
> This seems to be happening after peft@\`75808eb2a6e7b4c3ed8aec003b6eeb30a2db1495\`.
> 
> Normally \`adapter\_model.bin\` should be \> 16 MB. And while the 443 B \`adapter\_model.bin\` is loaded, the model behaves like not fine-tuned at all. In contrast, loading other checkpoints from the same training works as expected.
> 
> \`\`\`
> drwxrwxr-x 2 ubuntu ubuntu 4.0K Apr 9 12:55 .
> drwxrwxr-x 7 ubuntu ubuntu 4.0K Apr 9 12:54 ..
> \-rw-rw-r-- 1 ubuntu ubuntu 350 Apr 9 12:55 adapter\_config.json
> \-rw-rw-r-- 1 ubuntu ubuntu 443 Apr 9 12:55 adapter\_model.bin
> drwxr-xr-x 2 ubuntu ubuntu 4.0K Apr 9 12:06 checkpoint-400
> drwxr-xr-x 2 ubuntu ubuntu 4.0K Apr 9 12:06 checkpoint-600
> drwxr-xr-x 2 ubuntu ubuntu 4.0K Apr 9 12:07 checkpoint-800
> \`\`\`
> 
> I'm not sure if this is an issue to \`peft\` or not, or is this a duplication of other issues, but just leaving this for reference.
> 
> I've been testing with multiple versions of \`peft\`:
> 
> \* \`072da6d9d62\` works
> \* \`382b178911edff38c1ff619bbac2ba556bd2276b\` works
> \* \`75808eb2a6e7b4c3ed8aec003b6eeb30a2db1495\` not working
> \* \`445940fb7b5d38390ffb6707e2a989e89fff03b5\` not working
> \* \`1a6151b91fcdcc25326b9807d7dbf54e091d506c\` not working
> \* \`1117d4772109a098787ce7fc297cb6cd641de6eb\` not working
> 
> Steps to reproduce:
> 
> \`\`\`bash
> conda create python=3.8 -n test
> conda activate test
> git clone https://github.com/tloen/alpaca-lora.git
> cd alpaca-lora
> pip install -r requirements.txt
> 
> \# to workaround AttributeError: bitsandbytes/libbitsandbytes\_cpu.so: undefined symbol: cget\_col\_row\_stats
> cd /home/ubuntu/miniconda3/envs/test/lib/python3.8/site-packages/bitsandbytes/
> mv libbitsandbytes\_cpu.so libbitsandbytes\_cpu.so.bak
> cp libbitsandbytes\_cuda121.so libbitsandbytes\_cpu.so
> cd -
> conda install cudatoolkit
> 
> \# alpaca\_data\_cleaned\_first\_100.json is alpaca\_data\_cleaned.json with only the first 100 items, setting --val\_set\_size 0 because there're not enough data to build the test set
> python finetune.py --base\_model 'decapoda-research/llama-7b-hf' --data\_path '/data/datasets/alpaca\_data\_cleaned\_first\_100.json' --output\_dir './lora-alpaca' --val\_set\_size 0
> \`\`\`
> 
> \`\`\`bash
> $ ls -alh lora-alpaca
> total 16K
> drwxrwxr-x 2 ubuntu ubuntu 4.0K Apr 9 12:55 .
> drwxrwxr-x 7 ubuntu ubuntu 4.0K Apr 9 12:54 ..
> \-rw-rw-r-- 1 ubuntu ubuntu 350 Apr 9 12:55 adapter\_config.json
> \-rw-rw-r-- 1 ubuntu ubuntu 443 Apr 9 12:55 adapter\_model.bin
> \`\`\`
> 
> (\`adapter\_model.bin\` should normally be around 16 MB)
> 
> Running on Lambda Cloud A10 instance.

Basically this part seems to cause the weights to not be saved properly:

```auto
model.state_dict = (
    lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())
). __get__ (model, type(model))

model = torch.compile(model)

```

---

<div class="post-metadata">

**Author:** ![Suechun](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Suechun](https://discuss.huggingface.co/u/Suechun)\
**Post date:** [July 6, 2023, 3:19am UTC](https://discuss.huggingface.co/t/different-results-from-checkpoint-evaluation-when-loading-fine-tuned-llm-model/43106/3 "2023-07-06T03:19:07Z")

</div>

@ilektram Hi, can you share your script with finetuning llama, in my code when finetuning, there’s only training loss, no accuracy info. And although I trained for 3 epoches, the loss falls very slowly, after 3 epochs training, loss is still 0.96. I don’t know whta’s the problem.

---

<div class="post-metadata">

**Author:** ![Thinkcru](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thinkcru/32/13469_2.png) [@Thinkcru](https://discuss.huggingface.co/u/Thinkcru)\
**Post date:** [September 21, 2023, 4:03pm UTC](https://discuss.huggingface.co/t/different-results-from-checkpoint-evaluation-when-loading-fine-tuned-llm-model/43106/4 "2023-09-21T16:03:47Z")

</div>

Hey @00BER -

I was dealing with similar issue and I setup this gist to explain:

> <https://gist.github.com/cmosguy/4b5052507191200397c551fc24853ef0#file-03-ragntune-fine-tune-code-llama-py-L203-L209>

If you look at this you can see that it saves 443 sized files and are not changing at all from checkpoint to checkpoint.

I did this on the latest `peft` versions, do you have any ideas what can be done to resolve the issue?

---

<div class="post-metadata">

**Author:** ![00BER](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/00ber/32/16876_2.png) [@00BER](https://discuss.huggingface.co/u/00BER)\
**Post date:** [September 21, 2023, 6:02pm UTC](https://discuss.huggingface.co/t/different-results-from-checkpoint-evaluation-when-loading-fine-tuned-llm-model/43106/5 "2023-09-21T18:02:27Z")

</div>

@Thinkcru Not sure if it is the same issue, but for me commenting out that part solved the problem.

---

<div class="post-metadata">

**Author:** ![botkop](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/botkop/32/3143_2.png) [@botkop](https://discuss.huggingface.co/u/botkop)\
**Post date:** [September 22, 2023, 8:24am UTC](https://discuss.huggingface.co/t/different-results-from-checkpoint-evaluation-when-loading-fine-tuned-llm-model/43106/6 "2023-09-22T08:24:36Z")

</div>

probably related:

> [@Llama-2 Sequence Classification: Much lower accuracy on inference from checkpoint compared to model](https://discuss.huggingface.co/t/llama-2-sequence-classification-much-lower-accuracy-on-inference-from-checkpoint-compared-to-model/54910):
>
> I’m finetuning a Llama-2 sequence classification model with peft and qlora, and evaluating every 100 steps. I also save a checkpoint every 100 steps. When I load the checkpoint and do inference on the same validation set as during training, the accuracy is really much lower. Here’s the relevant code: Training: q\_config = BitsAndBytesConfig( load\_in\_4bit=True, bnb\_4bit\_quant\_type="nf4", bnb\_4bit\_use\_double\_quant=True, bnb\_4bit\_compute\_dtype=torch.bfloat16 ) model = AutoModelForSeq…