# Fine tuning using LOFTQ - CUDA out of memory error

**URL:** <https://discuss.huggingface.co/t/fine-tuning-using-loftq-cuda-out-of-memory-error/73532>\
**Category:** 🤗Transformers\
**Created:** [February 16, 2024, 10:07pm UTC](https://discuss.huggingface.co/t/fine-tuning-using-loftq-cuda-out-of-memory-error/73532 "2024-02-16T22:07:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![smallsuper](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@smallsuper](https://discuss.huggingface.co/u/smallsuper)\
**Post date:** [February 16, 2024, 10:07pm UTC](https://discuss.huggingface.co/t/fine-tuning-using-loftq-cuda-out-of-memory-error/73532/1 "2024-02-16T22:07:15Z")

</div>

I want to fine tune a quantized Mistral on a 24 GB GPU. but I am getting CUDA out of memory error. It seems the base\_model is not quantized.

```auto
from transformers import AutoModelForCausalLM
from peft import LoftQConfig, LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", device_map="auto")
loftq_config = LoftQConfig(loftq_bits=4)

lora_config = LoraConfig(
    init_lora_weights="loftq",
    loftq_config=loftq_config,
    r=16,
    lora_alpha=8,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

```

---

<div class="post-metadata">

**Author:** ![DenseLance](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/denselance/32/49696_2.png) [@DenseLance](https://discuss.huggingface.co/u/DenseLance)\
**Post date:** [February 17, 2024, 4:56am UTC](https://discuss.huggingface.co/t/fine-tuning-using-loftq-cuda-out-of-memory-error/73532/2 "2024-02-17T04:56:32Z")

</div>

When using LoftQ as a finetuning method, you would need to initialize the model first. Quantization is only done under the hood by `loftq_config`. On the other hand if you were to use QLoRA, you can quantize the model directly as a parameter under `from_pretrained` before finetuning.

---

<div class="post-metadata">

**Author:** ![smallsuper](https://avatars.discourse-cdn.com/v4/letter/s/4af34b/32.png) [@smallsuper](https://discuss.huggingface.co/u/smallsuper)\
**Post date:** [February 18, 2024, 4:40am UTC](https://discuss.huggingface.co/t/fine-tuning-using-loftq-cuda-out-of-memory-error/73532/3 "2024-02-18T04:40:49Z")

</div>

If we have to initialize the model before quantization, does this not defeat the purpose of quantization?

---

<div class="post-metadata">

**Author:** ![DenseLance](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/denselance/32/49696_2.png) [@DenseLance](https://discuss.huggingface.co/u/DenseLance)\
**Post date:** [February 18, 2024, 6:23am UTC](https://discuss.huggingface.co/t/fine-tuning-using-loftq-cuda-out-of-memory-error/73532/4 "2024-02-18T06:23:05Z")

</div>

Yep, which is why `LoftQConfig` was a confusing addition. You are meant to apply the LoftQ technique to a full-precision pre-trained weight first, as seen [here](https://github.com/yxli2123/LoftQ?tab=readme-ov-file#steps). From there on, you can quantize and save the model, so that in the future you would only need to load the quantized model.

My suggestion is to either use the already available “LoftQ-applied” models on HuggingFace, or stick to QLoRA.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [February 18, 2024, 6:23pm UTC](https://discuss.huggingface.co/t/fine-tuning-using-loftq-cuda-out-of-memory-error/73532/5 "2024-02-18T18:23:35Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.