# Training loop for LoRA

**URL:** <https://discuss.huggingface.co/t/training-loop-for-lora/106885>\
**Category:** 🤗Transformers\
**Created:** [September 16, 2024, 10:21am UTC](https://discuss.huggingface.co/t/training-loop-for-lora/106885 "2024-09-16T10:21:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![khatrimann](https://avatars.discourse-cdn.com/v4/letter/k/a6a055/32.png) [@khatrimann](https://discuss.huggingface.co/u/khatrimann)\
**Post date:** [September 16, 2024, 10:21am UTC](https://discuss.huggingface.co/t/training-loop-for-lora/106885/1 "2024-09-16T10:21:47Z")

</div>

Hey all, I wanted to ask if there is any way to apply LoRA on model and then train model in training loop without

```auto
SFTTrainer

```

I am Using the following way, is it fine?

```python
model_name = "meta-llama/Meta-Llama-3.1-8B-Instruct" # or any other model
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    # load_in_8bit=True,
    # quantization_config=quant_config,
    device_map="auto",
    max_memory={0: "4GiB", 1: "50GiB", "cpu": "0.0GiB"},
    offload_state_dict=True,
    low_cpu_mem_usage=True
)

...

model.config.use_cache = False
model.config.pretraining_tp = 1

model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)

peft_params = LoraConfig(
    r=32,
    lora_alpha=64,
    target_modules=[
        "q_proj",
        "v_proj",
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, peft_params)
model.print_trainable_parameters()
        
optimizer = Adam(model.parameters(), lr=1e-5)

for epoch_idx in [1]:
    for batch_idx, batch in tqdm(enumerate(train_data), total=len(train_data)):
        
        optimizer.zero_grad()
        outputs = model(**inputs)
        ...

```

And then finally using

```auto
merge_and_unload()

```

to merge adapter back to the model

---

<div class="post-metadata">

**Author:** ![BenjaminB](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/benjaminb/32/30898_2.png) [@BenjaminB](https://discuss.huggingface.co/u/BenjaminB)\
**Post date:** [September 17, 2024, 2:33pm UTC](https://discuss.huggingface.co/t/training-loop-for-lora/106885/2 "2024-09-17T14:33:24Z")

</div>

Yes, PEFT is not dependent on the usage of `Trainer` or `SFTTrainer`. Here is just one example of using LoRA with a custom training loop:

> <https://github.com/huggingface/peft/blob/main/examples/sequence_classification/LoRA.ipynb>

Depending on what you would like to do, custom training loops can get a bit complicated (think: mixed precision, gradient accumulation, etc.) but it’s absolutely possible.

---

<div class="post-metadata">

**Author:** ![khatrimann](https://avatars.discourse-cdn.com/v4/letter/k/a6a055/32.png) [@khatrimann](https://discuss.huggingface.co/u/khatrimann)\
**Post date:** [September 17, 2024, 4:21pm UTC](https://discuss.huggingface.co/t/training-loop-for-lora/106885/3 "2024-09-17T16:21:19Z")

</div>

Thanks mate. I hope they mention this somewhere the left or Lora page

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [September 18, 2024, 4:21am UTC](https://discuss.huggingface.co/t/training-loop-for-lora/106885/4 "2024-09-18T04:21:35Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.