# Help with merging LoRA weights back into base model :-)

**URL:** <https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968>\
**Category:** Beginners\
**Created:** [May 25, 2023, 2:04pm UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968 "2023-05-25T14:04:15Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmckinno](https://avatars.discourse-cdn.com/v4/letter/d/b38774/32.png) [@dmckinno](https://discuss.huggingface.co/u/dmckinno)\
**Post date:** [May 25, 2023, 2:04pm UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/1 "2023-05-25T14:04:16Z")

</div>

As best as I can tell, the LoraModel `merge_and_unload` attribute ([peft/lora.py at main · huggingface/peft · GitHub](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora.py#L343)) merges LoRA weights back into the main model.

However, I am having trouble getting a LoraModel type from my PeftModelForCausalLM. My current workflow is to define a pretrained model, define a LoraConfig, and use the `get_peft_model` function to being training. This works great, but I want to be able to merge the weights back into the base model and save.

My working assumption is that I need to either convert my PeftModelForCausalLM into a LoraModel or initialize the model as a LoraModel prior to training. However, when I copy the example in the LoraModel docstring ([peft/lora.py at main · huggingface/peft · GitHub](https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora.py#L103-L119)), I get an `TypeError` (TypeError: LoraModel. **init** () missing 1 required positional argument: ‘adapter\_name’). When I try passing a “lora” as a adapter name, I get another error.

I think that I am fundamentally thinking about this in the wrong way and would love some pointers. Both Google and Copilot chat have not been able to solve my problem.

---

<div class="post-metadata">

**Author:** ![dmckinno](https://avatars.discourse-cdn.com/v4/letter/d/b38774/32.png) [@dmckinno](https://discuss.huggingface.co/u/dmckinno)\
**Post date:** [May 25, 2023, 10:10pm UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/2 "2023-05-25T22:10:50Z")

</div>

I figured this out. The solution is quite simple.

A PeftModelForCausalLM actually inherits the LoraModel methods, so you can call `merged_model = merged.merge_and_unload()` to get back a base model with the LoRA weights applied.

My IDE would not autocomplete `merge_and_upload`, so I assumed the method wasn’t available. I still don’t need in the code where this method is inherited and would love for someone to point this out to me if feeling charitable.

---

<div class="post-metadata">

**Author:** ![Astropulse](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/astropulse/32/13830_2.png) [@Astropulse](https://discuss.huggingface.co/u/Astropulse)\
**Post date:** [June 20, 2023, 4:45pm UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/3 "2023-06-20T16:45:06Z")

</div>

Looking to get a model file from a base + lora myself. Can you explain in more detail how you were able to do it?

---

<div class="post-metadata">

**Author:** ![dmckinno](https://avatars.discourse-cdn.com/v4/letter/d/b38774/32.png) [@dmckinno](https://discuss.huggingface.co/u/dmckinno)\
**Post date:** [June 22, 2023, 1:48am UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/4 "2023-06-22T01:48:56Z")

</div>

Try this. Basic steps are to:  
1/ load the base model  
2/ train the base model  
3/ save the LoRA adapter  
4/ reload the base model at half/full precision  
5/ merge the LoRA weights with the base model  
6/ save

base\_model = AutoModelForCausalLM.from\_pretrained(“base\_model”, load\_in\_8bit=True, torch\_dtype=torch.float16, device\_map=“auto”)

base\_model = prepare\_model\_for\_int8\_training(base\_model)

peft\_model = get\_peft\_model(base\_model, peft\_config)

training\_args = TrainingArguments()  
trainer = Trainer()  
trainer.train()

peft\_model.save\_pretrained(lora\_adapter, save\_adapter=True, save\_config=True)

model\_to\_merge = PeftModel.from\_pretrained(AutoModelForCausalLM.from\_pretrained(base\_model).to(“cuda”), lora\_adapter)

merged\_model = model\_to\_merge.merge\_and\_unload()  
merged\_model.save\_pretrained(merged\_model)

---

<div class="post-metadata">

**Author:** ![juaramos](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/juaramos/32/17831_2.png) [@juaramos](https://discuss.huggingface.co/u/juaramos)\
**Post date:** [July 22, 2023, 7:42am UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/5 "2023-07-22T07:42:46Z")

</div>

These are the correct steps to create a model version form a base model and my train?

---

<div class="post-metadata">

**Author:** ![ssm1990](https://avatars.discourse-cdn.com/v4/letter/s/85e7bf/32.png) [@ssm1990](https://discuss.huggingface.co/u/ssm1990)\
**Post date:** [September 15, 2023, 1:17pm UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/6 "2023-09-15T13:17:39Z")

</div>

A related question, why the model size in disk is almost doubling after merging even when the number of parameters remaining the same?

---

<div class="post-metadata">

**Author:** ![accOne996795](https://avatars.discourse-cdn.com/v4/letter/a/f6c823/32.png) [@accOne996795](https://discuss.huggingface.co/u/accOne996795)\
**Post date:** [January 21, 2024, 5:32am UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/7 "2024-01-21T05:32:31Z")

</div>

But when I try to save the model weights on model\_to\_merge then I still get only the adapter safetensors and not the safetensors for the entire model.  
How do i get that?

---

<div class="post-metadata">

**Author:** ![Abhinav28](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@Abhinav28](https://discuss.huggingface.co/u/Abhinav28)\
**Post date:** [January 31, 2024, 9:25am UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/8 "2024-01-31T09:25:53Z")

</div>

I have the same question, why the size difference in the base model and merged model?

---

<div class="post-metadata">

**Author:** ![ferrazzipietro](https://avatars.discourse-cdn.com/v4/letter/f/b5e925/32.png) [@ferrazzipietro](https://discuss.huggingface.co/u/ferrazzipietro)\
**Post date:** [April 29, 2024, 2:48pm UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/9 "2024-04-29T14:48:48Z")

</div>

I see 2 potential reasons for that, unsure if any of them is applicable to your use case as I cannot see your code.

1. The merged model (base model + LoRA adapters) has the numer of parameters of the base model + the number of parameters of the inserted LoRA adapters. For this reason,`# params merged_model > # params base_model`, therefore the increase in size

2. When reloading the model, make sure to provide the same data type as in training to ensure the same size is mantained. For example, if the trained model was loaded in half precision, also the model\_to\_merge should be loaded in half precision to enforce comparable size e.g:

```auto
model_to_train = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16) 
...
model_to_merge = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(base_model, torch_dtype=torch.bfloat16), lora_adapter)

```

In the example provided by @dmckinno, the `model_to_train` is loaded in 8 bits (`load_in_8bit=True`), and then `model_to_merge` is loaded in full precision, since no parameter is provided and the default will be used (float32)

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [May 1, 2024, 2:33pm UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/10 "2024-05-01T14:33:04Z")

</div>

Hi,

I explain it in more detail : [Config.json is not saving after finetuning Llama 2 - #6 by nielsr](https://discuss.huggingface.co/t/config-json-is-not-saving-after-finetuning-llama-2/56871/6), hope you find it useful!

And this is a bit related: [Further finetuning a LoRA finetuned CausalLM Model - #4 by nielsr](https://discuss.huggingface.co/t/further-finetuning-a-lora-finetuned-causallm-model/36987/4)

---

<div class="post-metadata">

**Author:** ![stefutz101](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stefutz101/32/28419_2.png) [@stefutz101](https://discuss.huggingface.co/u/stefutz101)\
**Post date:** [June 20, 2024, 8:13am UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/11 "2024-06-20T08:13:40Z")

</div>

Wasn’t here missing the part where you can push this into HUB?

```auto
merged_model.push_to_hub("repo_id")
tokenizer.push_to_hub("repo_id")

```

---

<div class="post-metadata">

**Author:** ![P1sc3s007](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/p1sc3s007/32/30377_2.png) [@P1sc3s007](https://discuss.huggingface.co/u/P1sc3s007)\
**Post date:** [February 6, 2025, 7:10am UTC](https://discuss.huggingface.co/t/help-with-merging-lora-weights-back-into-base-model/40968/13 "2025-02-06T07:10:03Z")

</div>

why my ‘prepare\_model\_for\_int8\_training’ is not defined?

```auto
Traceback (most recent call last):
  File "/root/autodl-tmp/train_pt.py", line 67, in <module>
    model = prepare_model_for_int8_training(model)
NameError: name 'prepare_model_for_int8_training' is not defined

```

```python
from transformers import *
from peft import *
import torch
from datasets import load_dataset
import os
from torch.utils.data import DataLoader
from transformers import default_data_collator, get_linear_schedule_with_warmup
from tqdm import tqdm
from datasets import load_dataset
from tensorboard import * 

device = "cuda"
tokenizer_name_or_path = "LLM4Binary/llm4decompile-1.3b-v1.5"
model_name_or_path = "LLM4Binary/llm4decompile-1.3b-v1.5"
dataset_name = "asm2c"
text_column = "asm text"
label_column = "text_label"
max_length = 64
lr = 3e-2
num_epochs = 50
batch_size = 8

from datasets import load_dataset

dataset = load_dataset("json", data_files="./traindata.jsonl")
dataset = dataset["train"].train_test_split(0.2)

tokenizer = AutoTokenizer.from_pretrained("LLM4Binary/llm4decompile-1.3b-v1.5")

def preprocess_function(examples):
    inputs = examples["input"]
    outputs = examples["output"]

    # 合并input和output列
    merged_texts = [f"{input} {output_text}" for input, output_text in zip(inputs, outputs)]
    
    model_inputs = tokenizer(merged_texts, truncation=True, padding="max_length", max_length=512)
    model_inputs["labels"] = model_inputs["input_ids"].copy() # 设置labels
    return model_inputs

processed_datasets = dataset.map(
    preprocess_function,
    batched=True,
    num_proc=1,
    remove_columns=dataset["train"].column_names,
    load_from_cache_file=False,
    desc="Running tokenizer on dataset",
)

train_dataset = processed_datasets["train"]
eval_dataset = processed_datasets["test"]

peft_config = PromptTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    prompt_tuning_init=PromptTuningInit.TEXT,
    num_virtual_tokens=8,
    prompt_tuning_init_text="What's the souce code of this asm?",
    tokenizer_name_or_path=model_name_or_path,
)
checkpoint_name = f"{dataset_name}_{model_name_or_path}_{peft_config.peft_type}_{peft_config.task_type}_v1.pt".replace(
    "/", "_"
)

# creating model
model = AutoModelForCausalLM.from_pretrained("LLM4Binary/llm4decompile-1.3b-v1.5", load_in_8bit=True, torch_dtype=torch.float16, device_map="auto")
model = prepare_model_for_int8_training(model)
peft_model = get_peft_model(model, peft_config)

training_args = TrainingArguments(
    output_dir="./results3", # 保存模型的目录
    evaluation_strategy="epoch", # 每个 epoch 进行评估
    save_strategy="epoch", # 每个 epoch 结束时保存模型              
    learning_rate=2e-5,
    per_device_train_batch_size=4, # 训练时的batch_size
    per_device_eval_batch_size=8, # 验证时的batch_size
    logging_steps=10, # log 打印的频率
    num_train_epochs=3,
    weight_decay=0.01,
    load_best_model_at_end=False
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    #data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True)
)

trainer.train()
'''
trainer.evaluate(eval_dataset)

# 训练结束后手动保存模型
trainer.save_model(output_dir="./tuned_model") # 保存最终的模型到指定的目录
tokenizer.save_pretrained(save_directory="./tuned_tokenizer") # 保存tokenizer
'''
lora_adapter = "./lora_adapter"
peft_model.save_pretrained(lora_adapter, save_adapter=True, save_config=True)

model_to_merge = PeftModel.from_pretrained(AutoModelForCausalLM.from_pretrained(model_name_or_path).to("cuda"), lora_adapter)

merged_model = model_to_merge.merge_and_unload()
merged_model.save_pretrained(merged_model)

```