# Cannot Merge Lora weights back to the base model **URL:** \ **Category:** Intermediate\ **Created:** [October 29, 2024, 2:08pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562 "2024-10-29T14:08:40Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![luizaMici](https://avatars.discourse-cdn.com/v4/letter/l/6f9a4e/32.png) [@luizaMici](https://discuss.huggingface.co/u/luizaMici)\ **Post date:** [October 29, 2024, 2:08pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562/1 "2024-10-29T14:08:40Z")
I have a Flux-Dev base model which has been trained with the lora technique using the SimpleTuner framework ([SimpleTuner/documentation/quickstart/FLUX.md at main · bghira/SimpleTuner · GitHub](https://github.com/bghira/SimpleTuner/blob/main/documentation/quickstart/FLUX.md)). The training process has generated checkpoint folders containing the following files: - optimizer.bin - pytorch\_lora\_weights.safetensors - scheduler.bin - random\_states\_0.pkl - training\_state-mydataset.json - training\_state.json I want to load the Flux base model and the lora weights from one of the checkpoints, and merge the lora weights into the base model. Following the Lora documentation here [LoRA](https://huggingface.co/docs/peft/main/en/developer_guides/lora) I tried the following: ` from peft import PeftModel from diffusers import FluxTransformer2DModel lora\_path =\ base\_model = FluxTransformer2DModel.from\_pretrained(\, subfolder=‘transformer’, ) model = PeftModel.from\_pretrained(base\_model, lora\_path) model.merge\_and\_unload() ` I get the following error: ValueError: Can’t find ‘adapter\_config.json’ at \’ There is indeed no such file in there. How can I load the lora weights and merge them to the model in this case? Is there another way? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [October 29, 2024, 2:31pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562/2 "2024-10-29T14:31:22Z")
Try this. ```py #lora_path = lora_path = # lora_path ="pytorch_lora_weights.safetensors" # for example ``` ---
**Author:** ![luizaMici](https://avatars.discourse-cdn.com/v4/letter/l/6f9a4e/32.png) [@luizaMici](https://discuss.huggingface.co/u/luizaMici)\ **Post date:** [October 29, 2024, 2:38pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562/3 "2024-10-29T14:38:56Z")
Thanks for the answer John. I tried your suggestion and I get the same error again: ValueError: Can’t find ‘adapter\_config.json’ at \ ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [October 29, 2024, 2:43pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562/4 "2024-10-29T14:43:20Z")
**from\_pretrained()** is a function that reads the **HF repo format**. For single files, use **LoraModel class** or **load\_lora\_weights() from pipeline**. > **[LoRA](https://huggingface.co/docs/peft/package_reference/lora)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[Load adapters](https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ```auto pipe.load_lora_weights( pretrained_model_name_or_path_or_dict="loras", weight_name="lora.safetensors" ) ``` ---
**Author:** ![luizaMici](https://avatars.discourse-cdn.com/v4/letter/l/6f9a4e/32.png) [@luizaMici](https://discuss.huggingface.co/u/luizaMici)\ **Post date:** [October 29, 2024, 2:52pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562/5 "2024-10-29T14:52:48Z")
I have tried using the pipeline **load\_lora\_weights()** as well and that works great for running inference with the pipeline. However, I do not see an obvious way how to merge the LoRA weights into the base model to be able to use just the merged model for example for further training on different datasets. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [October 29, 2024, 2:57pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562/6 "2024-10-29T14:57:30Z")
```py from pathlib import Path a_list = [] w_list = [] for lora in lora_file_list: new_lora_file = lora w_name = Path(new_lora_file).name a_name = Path(new_lora_file).stem pipe.load_lora_weights(new_lora_file, weight_name=w_name, adapter_name=a_name, low_cpu_mem_usage=False) a_list.append(a_name) w_list.append(1.0) pipe.set_adapters(a_list, adapter_weights=w_list) pipe.fuse_lora(adapter_names=a_list, lora_scale=1.0) pipe.unload_lora_weights() pipe.save_pretrained("lora_merged_model") ``` If you want to apply LoRA only to the transformer, I think you can just torch.load\_file with the safetensors library as just a torch model, then apply LoRA with PEFT and save\_file. ---
**Author:** ![luizaMici](https://avatars.discourse-cdn.com/v4/letter/l/6f9a4e/32.png) [@luizaMici](https://discuss.huggingface.co/u/luizaMici)\ **Post date:** [October 29, 2024, 4:08pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562/7 "2024-10-29T16:08:14Z")
I do indeed want to apply LoRA only to the transformer. Here is what I got so far: ```auto import safetensors lora_weights_file = lora_weights_dict = safetensors.torch.load_file(lora_weights_file) ``` Now I have a dictionary of the LoRA weights. How can I apply the LoRA weights with PEFT to the model? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [October 29, 2024, 11:59pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562/8 "2024-10-29T23:59:43Z")
> **[LoRA](https://huggingface.co/docs/peft/v0.9.0/en/package_reference/lora#peft.LoraModel)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. Sorry, I made a mistake. Apparently you don’t have to load up to state\_dict, just the normal model. ```py base_model = FluxTransformer2DModel.from_pretrained(, subfolder=‘transformer’, ) ``` Now all you should have to do is set up LoraConfig and do get\_peft\_model(), but I don’t know the proper contents of LoraConfig in this case. Usually Pipeline internal does it on its own… > **[GitHub - huggingface/peft: 🤗 PEFT: State-of-the-art Parameter-Efficient...](https://github.com/huggingface/peft?tab=readme-ov-file#quickstart)** > > 🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning. Edit: I found it. > **[GitHub - haofanwang/Lora-for-Diffusers: The most easy-to-understand tutorial for using...](https://github.com/haofanwang/Lora-for-Diffusers)** > > The most easy-to-understand tutorial for using LoRA (Low-Rank Adaptation) within diffusers framework for AI Generation Researchers🔥 ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [January 21, 2025, 2:57pm UTC](https://discuss.huggingface.co/t/cannot-merge-lora-weights-back-to-the-base-model/114562/9 "2025-01-21T14:57:54Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.