# Missing trainable parameters in a loaded LoRA model

**URL:** <https://discuss.huggingface.co/t/missing-trainable-parameters-in-a-loaded-lora-model/45816>\
**Category:** 🤗Transformers\
**Created:** [July 6, 2023, 2:17am UTC](https://discuss.huggingface.co/t/missing-trainable-parameters-in-a-loaded-lora-model/45816 "2023-07-06T02:17:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![adamxyang](https://avatars.discourse-cdn.com/v4/letter/a/76d3ee/32.png) [@adamxyang](https://discuss.huggingface.co/u/adamxyang)\
**Post date:** [July 6, 2023, 2:17am UTC](https://discuss.huggingface.co/t/missing-trainable-parameters-in-a-loaded-lora-model/45816/1 "2023-07-06T02:17:30Z")

</div>

I followed [this blog post](https://huggingface.co/blog/peft) for saving and loading a PEFT model. I defined my model as

```auto
model = AutoModelForSequenceClassification.from_pretrained('roberta-base', return_dict=True)
peft_config = LoraConfig(task_type="SEQ_CLS", inference_mode=False, r=8, lora_alpha=16, lora_dropout=0.1)
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

```

I saved it with

```auto
save_path = './test_save'
model.save_pretrained(save_path)

```

and load it with

```auto
config = PeftConfig.from_pretrained(save_path)
model = AutoModelForSequenceClassification.from_pretrained(config.base_model_name_or_path)
model = PeftModel.from_pretrained(model, save_path)
model.print_trainable_parameters()

```

However, I got 300k less trainable parameters when loading the same model. When saving the model, it prints

```auto
trainable params: 1479172 || all params: 125534212 || trainable%: 1.1783018959006968

```

after loading, it prints

```auto
trainable params: 1184260 || all params: 125534212 || trainable%: 0.9433762964951737

```

Is this the correct way of saving and loading a PEFT model?

---

<div class="post-metadata">

**Author:** ![adamxyang](https://avatars.discourse-cdn.com/v4/letter/a/76d3ee/32.png) [@adamxyang](https://discuss.huggingface.co/u/adamxyang)\
**Post date:** [July 6, 2023, 5:37pm UTC](https://discuss.huggingface.co/t/missing-trainable-parameters-in-a-loaded-lora-model/45816/2 "2023-07-06T17:37:45Z")

</div>

Found what’s wrong, `model.save_pretrained` somehow automatically freezes all LoRA parameters and only keep the last one or two fully connected classifier head layers trainable (didn’t see this behaviour documented), so we just need to set all LoRA parameters trainable again with:

```auto
for name, param in model.named_parameters():
    if 'lora' in name:
        param.requires_grad = True

```