# Loading an LoRA adapter trained on quantized model on a non-quantized model

**URL:** <https://discuss.huggingface.co/t/loading-an-lora-adapter-trained-on-quantized-model-on-a-non-quantized-model/61403>\
**Category:** Intermediate\
**Created:** [November 7, 2023, 3:57pm UTC](https://discuss.huggingface.co/t/loading-an-lora-adapter-trained-on-quantized-model-on-a-non-quantized-model/61403 "2023-11-07T15:57:11Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![datadoktergroen](https://avatars.discourse-cdn.com/v4/letter/d/c5a1d2/32.png) [@datadoktergroen](https://discuss.huggingface.co/u/datadoktergroen)\
**Post date:** [November 7, 2023, 3:57pm UTC](https://discuss.huggingface.co/t/loading-an-lora-adapter-trained-on-quantized-model-on-a-non-quantized-model/61403/1 "2023-11-07T15:57:11Z")

</div>

Imagine the following (common) training-inference scenario:

**Training**

- You load a LLama-2 model in 4 bit and you train using PEFT (Q-LoRA)
- You train an adapter and push it to the hub

**Inference options:**  
**Option 1: Use Quantized base model w/ adapter** :  
It seems the most logical, since the adapter was calibrated on a model with this precision. However, doing this throws the error:

```auto
RuntimeError: expected scalar type Float but found Half

```

**Option 2: use Non-quantized base model w/ adapter**  
Loading the non-quantized model does work, but wouldn’t using the same adapter on a higher-precision model cause the adapter to be miscalibrated?

```auto
model_name = "meta-llama/Llama-2-13b-hf"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    quantization_config=bnb_config, # This line apparently causes an error?
    device_map='auto'
)
model.config.use_cache = False
model.load_adapter(model_name)

```

Could somebody help me understand this?