# Creating language model only Lora Config

**URL:** <https://discuss.huggingface.co/t/creating-language-model-only-lora-config/170186>\
**Category:** 🤗Transformers\
**Created:** [November 10, 2025, 12:58pm UTC](https://discuss.huggingface.co/t/creating-language-model-only-lora-config/170186 "2025-11-10T12:58:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![StevenLimcorn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stevenlimcorn/32/38800_2.png) [@StevenLimcorn](https://discuss.huggingface.co/u/StevenLimcorn)\
**Post date:** [November 10, 2025, 12:58pm UTC](https://discuss.huggingface.co/t/creating-language-model-only-lora-config/170186/1 "2025-11-10T12:58:44Z")

</div>

I am currently looking into doing SFT fine-tuning of a VLLM using this guide provided: [Fine-tuning a Multimodal Model Using SFT (Single or Multi-Image Dataset)](https://huggingface.co/docs/trl/main/en/training_vlm_sft) , and I realised that the Lora config mentioned is just using the “all-linear” target\_modules like such:

```python
peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.05,
    r=16,
    bias="none",
    target_modules="all-linear",
    task_type="CAUSAL_LM",
    modules_to_save=[
        "lm_head",
        "embed_tokens",
    ],
)

```

Just curious, is there a way to apply the target\_modules to these

```auto
["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"]

```

and I only want it to be effected to only the language model layers?

Currently, I tried doing this:

```python
model = "google/gemma-3-4b-it"
layer_pattern_name = "model.language_model.layers"
peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.05,
    r=16,
    bias="none",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    layers_to_transform = [i for i in range(len(eval(layer_pattern_name)))],
    layer_pattern_name=layer_pattern_name,
    task_type="CAUSAL_LM"
)

```

But I got this error message when passing this config to SFTTrainer

ValueError: Target modules {‘q\_proj’, ‘up\_proj’, ‘down\_proj’, ‘gate\_proj’, ‘o\_proj’, ‘v\_proj’, ‘k\_proj’} not found in the base model. Please check the target modules and try again. Note: You specified ‘layers\_to\_transform’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]. You also specified ‘layers\_pattern’: model.language\_model.layers.

Wondering if anyone has had the same issue. Thank you!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 10, 2025, 1:36pm UTC](https://discuss.huggingface.co/t/creating-language-model-only-lora-config/170186/2 "2025-11-10T13:36:42Z")

</div>

The module name to be specified seems to differ slightly from other models.

* * *

You can apply LoRA to `["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"]` and restrict it to the **text** stack only. Your error comes from two things: using the wrong key (`layers_pattern` is the correct field) and relying on a brittle module path. On Gemma-3 VLMs the **vision** tower also contains `q_proj/k_proj/v_proj/o_proj` names, so you must scope LoRA to the text layers explicitly. ([Hugging Face](https://huggingface.co/docs/peft/main/en/conceptual_guides/lora "LoRA"))

# Background in one page

- **How PEFT picks modules.** `target_modules` matches by module name (suffix or regex). If those names appear in multiple submodules, PEFT will hit all of them unless you constrain by layer indices and a layer pattern. That’s what `layers_to_transform` and `layers_pattern` do. ([Hugging Face](https://huggingface.co/docs/peft/en/package_reference/lora "LoRA"))
- **Why “LM-only” matters for VLMs.** Gemma-3 VLM uses a SigLIP vision encoder. That encoder defines `q_proj/k_proj/v_proj/out_proj` too, so a naive `target_modules=[...]` would also touch vision unless you filter for the text stack. ([Google Developers Blog](https://developers.googleblog.com/en/gemma-explained-whats-new-in-gemma-3/ "Gemma explained: What's new in Gemma 3"))
- **Text stack shape and naming drift.** Gemma-3 configs expose the text stack via `config.text_config.num_hidden_layers`. Internal module paths changed across releases, so hard-coding `model.language_model.layers` is fragile. Use a pattern that matches both `model.layers` and `language_model.layers`. The refactor is visible in user reports about adapter key paths moving. ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/gemma3 "Gemma 3"))
- **vLLM and TRL context.** TRL’s SFT VLM guide is the right procedure for Gemma-3 multimodal SFT. vLLM docs also show the familiar q/k/v and MLP projection names. ([Hugging Face](https://huggingface.co/docs/trl/main/en/training_vlm_sft "Fine-tuning a Multimodal Model Using SFT (Single or Multi- ..."))

# Working configurations

## A) “LM-only” LoRA with explicit layer filtering

This confines LoRA to the text stack even on multimodal checkpoints and across Transformers versions.

```python
# deps: peft>=0.11, transformers>=4.52 (Gemma 3 supported)
# URLs:
# - PEFT Lora fields: https://huggingface.co/docs/peft/en/conceptual_guides/lora
# - TRL VLM SFT: https://huggingface.co/docs/trl/main/en/training_vlm_sft
from peft import LoraConfig
from transformers import AutoModelForCausalLM

model_id = "google/gemma-3-4b-it"
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Read the number of text layers from config to avoid guessing paths
n_text_layers = getattr(getattr(model.config, "text_config", None), "num_hidden_layers", None)
if n_text_layers is None:
    # Fallback: common single-stack configs still expose num_hidden_layers at top-level
    n_text_layers = getattr(model.config, "num_hidden_layers")

peft_config = LoraConfig(
    r=16,
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    # Constrain to the text transformer blocks only
    layers_to_transform=list(range(n_text_layers)),
    # Accept both pre/post-refactor names:
    layers_pattern=r"(?:^|\.)(model|language_model)\.layers$",
    # Optional: keep embeddings/head in full precision
    modules_to_save=["lm_head", "embed_tokens"],
)

```

Why this works:

- `layers_to_transform` tells PEFT which block indices to touch.
- `layers_pattern` tells PEFT which **stack** owns those indices. The regex above matches either `...model.layers` or `...language_model.layers`, covering recent naming changes. ([Hugging Face](https://huggingface.co/docs/peft/main/en/conceptual_guides/lora "LoRA"))

## B) Minimal change if you fine-tune a **text-only** Gemma-3

If your checkpoint is **not** multimodal, you can omit the layer filtering and just list the projections. Many community LoRAs use exactly these targets. For VLMs, prefer Option A. ([Kaggle](https://www.kaggle.com/code/heidichoco/gemma-fine-tuning-for-beginners-with-huggingface "Gemma Fine-tuning for Beginners with Huggingface"))

```python
peft_config = LoraConfig(
    r=16, lora_alpha=16, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
    modules_to_save=["lm_head", "embed_tokens"],
)

```

# Quick self-check and debugging

Run these once before training:

```python
# 1) Confirm the text stack path and layer count
cands = [n for n,_ in model.named_modules() if n.endswith(".layers")]
print("layer containers:", cands) # should include ...model.layers or ...language_model.layers

# 2) List modules PEFT would touch with your targets
targets = ("q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj")
hits = [n for n,_ in model.named_modules() if n.endswith(targets)]
print("sample hits:", hits[:10])

```

If you still see “Target modules not found,” it means your names or pattern do not match what the model exposes. This is a common PEFT error when paths differ, or when `layers_pattern` is wrong. Adjust the pattern to your actual container path. ([GitHub](https://github.com/huggingface/peft/issues/1206 "ERROR ”Target modules not found in the base model ..."))

# Notes that avoid future breakage

- Use **`layers_pattern`** (correct key). `layer_pattern_name` is not a valid field. PEFT documents `layers_pattern` and `layers_to_transform` as the supported scoping knobs. ([Hugging Face](https://huggingface.co/docs/peft/main/en/conceptual_guides/lora "LoRA"))
- Derive the layer count from `config.text_config.num_hidden_layers` when present. Gemma-3 VLM configs expose this under `text_config`. ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/gemma3 "Gemma 3"))
- Expect model-internal renames over time. Users report LoRA adapter paths moving across refactors. Let the regex handle both `model.layers` and `language_model.layers`. ([GitHub](https://github.com/huggingface/transformers/issues/38927 "Can't load my LoRA checkpoint after gemma3 refactor"))
- Multimodal models: the vision encoder also uses `q_proj/k_proj/v_proj/out_proj`. Do not rely on `target_modules` alone for LM-only tuning. Constrain with `layers_pattern` + `layers_to_transform`. ([Hugging Face](https://huggingface.co/HuggingFaceM4/tiny-random-siglip/blob/main/modeling_siglip.py "modeling\_siglip.py · HuggingFaceM4/tiny-random-siglip at main"))
- TRL VLM SFT is the recommended recipe for Gemma-3 multimodal SFT; pass the `peft_config` directly to `SFTTrainer`. ([Hugging Face](https://huggingface.co/docs/trl/main/en/training_vlm_sft "Fine-tuning a Multimodal Model Using SFT (Single or Multi- ..."))

# Why these particular targets

The list covers attention projections and the MLP bottleneck. This is the standard set for Llama-family models and works well for Gemma-3 text blocks. You can also start with attention-only (`q/k/v/o`) if you need smaller adapters. vLLM’s Gemma mapping shows the same projections and groups like `qkv_proj` and `gate_up_proj`. ([VLLM Docs](https://docs.vllm.ai/en/stable/api/vllm/model_executor/models/gemma.html "gemma - vLLM"))

# Short, vetted extras

**Guides**

- TRL: Fine-tuning a Multimodal Model using SFT. Clear end-to-end VLM SFT walkthrough. ([Hugging Face](https://huggingface.co/docs/trl/main/en/training_vlm_sft "Fine-tuning a Multimodal Model Using SFT (Single or Multi- ..."))
- Blog: Fine-tune VLMs with TRL (conceptual + code). ([Phil Schmid](https://www.philschmid.de/fine-tune-multimodal-llms-with-trl "How to Fine-Tune Multimodal Models or VLMs ..."))

**APIs and docs**

- PEFT LoRA reference and scoping fields. Concise definitions and behavior. ([Hugging Face](https://huggingface.co/docs/peft/en/package_reference/lora "LoRA"))
- Transformers Gemma-3 docs. Config fields and text stack metadata. ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/gemma3 "Gemma 3"))

**Examples**

- Community Gemma-3 LoRA using the same targets. Quick sanity check. ([Kaggle](https://www.kaggle.com/code/heidichoco/gemma-fine-tuning-for-beginners-with-huggingface "Gemma Fine-tuning for Beginners with Huggingface"))

---

<div class="post-metadata">

**Author:** ![StevenLimcorn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stevenlimcorn/32/38800_2.png) [@StevenLimcorn](https://discuss.huggingface.co/u/StevenLimcorn)\
**Post date:** [November 10, 2025, 9:58pm UTC](https://discuss.huggingface.co/t/creating-language-model-only-lora-config/170186/3 "2025-11-10T21:58:33Z")

</div>

Thank you so much for the solution @John6666 . This is super helpful. I had a go with passing the exact same peft\_config to the SFTTrainer, and it seems like I am still getting the same issue here:

```auto
ValueError: No modules were targeted for adaptation. This might be caused by a combination of mismatched target modules and excluded modules. Please check your `target_modules` and `exclude_modules` configuration. You may also have only targeted modules that are marked to be saved (`modules_to_save`). Note: You specified 'layers_to_transform': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]. You also specified 'layers_pattern': (?:^|\.)(model|language_model)\.layers$.

```

For context I am using these versions

```auto
peft>=0.16.0
transformers>=4.56.2

```

---

<div class="post-metadata">

**Author:** ![StevenLimcorn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stevenlimcorn/32/38800_2.png) [@StevenLimcorn](https://discuss.huggingface.co/u/StevenLimcorn)\
**Post date:** [November 10, 2025, 11:54pm UTC](https://discuss.huggingface.co/t/creating-language-model-only-lora-config/170186/4 "2025-11-10T23:54:43Z")

</div>

Update: just got the solution by using this pattern:

```auto
r"(?:language_model|model)\.layers"

```

This allows only language\_model to be filtered down