# Multimodal Prefix Caching with Qwen3-VL

**URL:** <https://discuss.huggingface.co/t/multimodal-prefix-caching-with-qwen3-vl/170849>\
**Category:** Models\
**Created:** [November 25, 2025, 5:30pm UTC](https://discuss.huggingface.co/t/multimodal-prefix-caching-with-qwen3-vl/170849 "2025-11-25T17:30:08Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![glasbruch](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/glasbruch/32/56035_2.png) [@glasbruch](https://discuss.huggingface.co/u/glasbruch)\
**Post date:** [November 25, 2025, 5:30pm UTC](https://discuss.huggingface.co/t/multimodal-prefix-caching-with-qwen3-vl/170849/1 "2025-11-25T17:30:08Z")

</div>

Hello everyone,

I am experimenting with prefix caching in **Qwen3VL** for multimodal prompts. Specifically, I have a prefix prompt that includes demo images, which I want to cache and later reuse with a separate multimodal query prompt.

Here’s a simple example setup (see code snippet below):

- The **prefix** contains a cached image.

- The **query prompt** contains another image and a text prompt.

However, the model fails to process the image from the query prompt correctly. I suspect this is related to how position embeddings are handled when using caching. While prefix caching works fine for text-only prompts, it seems that for image inputs, Qwen3VL’s multimodal RoPE is not being applied correctly. As a result, the model cannot distinguish between the two images.

I’ve noticed similar behavior in other MLLMs, such as **Gemma3** , where the model responds by stating that the two images are identical.

Has anyone encountered this issue or found a workaround for properly using prefix caching with multimodal prompts?

```auto
model_id = "Qwen/Qwen3-VL-8B-Instruct"
    model = Qwen3VLForConditionalGeneration.from_pretrained(
        model_id,
        dtype=torch.bfloat16,
        attn_implementation="flash_attention_2",
        device_map="auto"
    )
    processor = AutoProcessor.from_pretrained(model_id)

    messages = [{"role": "user", "content": [{"type": "image", "image": img1}]}]

    inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=False,
        return_dict=True,
        return_tensors="pt"
    ).to(model.device)
    
    with torch.no_grad():
        prefix_output = model(**inputs, use_cache=True)

    prompt_cache = prefix_output.past_key_values
    
    # Now create the full prompt with both image and text
    prompt = "Compare the images."
    messages = [{"role": "user", "content": [
        {"type": "image", "image": img2},
        {"type": "text", "text": prompt}
    ]}]
    
    new_inputs = processor.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=True,
        return_dict=True,
        return_tensors="pt"
    ).to(model.device)

    # Manually set cache_position
    cache_length = prompt_cache[0][0].shape[2]
    new_seq_length = new_inputs["input_ids"].shape[1]
    cache_position = torch.arange(cache_length, cache_length + new_seq_length, device=model.device)

    # Use cache_position to skip the cached part
    past_key_values = copy.deepcopy(prompt_cache)

    outputs = model.generate(
        **new_inputs,
        past_key_values=past_key_values,
        use_cache=True,
        cache_position=cache_position,
        max_new_tokens=512
        )
    
    print(processor.batch_decode(outputs, skip_special_tokens=True)[0])

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 25, 2025, 10:44pm UTC](https://discuss.huggingface.co/t/multimodal-prefix-caching-with-qwen3-vl/170849/2 "2025-11-25T22:44:45Z")

</div>

Seems current limitation. Or rather just a bug…?

* * *

This is a known limitation/bug pattern in current Qwen-VL and similar MLLMs (including Gemma-3). What you are attempting – “cached multimodal prefix (with image A) + new multimodal query (with image B) in a later call using `past_key_values`” – is essentially not supported by the shipped Hugging Face implementations. The failure you see is not a simple bug in your code; it is how the models’ caching and multimodal RoPE are wired.

Below is a detailed breakdown of why, plus realistic workarounds and what would be required to “really” fix it.

* * *

## 1. What you are trying to do (rephrased)

Your experiment:

1. First call (prefix):

2. Second call (query):

Intended semantics:

- Treat the first call as a **cached demo prefix** containing `img1`.
- Treat the second call as “demo prefix (from cache) + new image `img2` + text prompt”, without recomputing the demo.

Observed behavior:

- The model largely ignores `img2` or treats `img1` and `img2` as identical.
- This matches what you see in Gemma-3 (second image effectively ignored).

This behavior is exactly what one would expect given how Qwen-VL-style models handle multimodal RoPE and caching.

* * *

## 2. How Qwen-VL-style models actually handle multimodal + caching

### 2.1 Prefill vs decode phases

Internally, Qwen2.5-VL and Qwen3-VL distinguish two phases:

1. **Prefill phase** (first call of a generation):

2. **Decode phase** (subsequent steps with non-empty cache):

This “multimodal prefill + text decode” design shows up throughout the Qwen-VL code and documentation. The Qwen3-VL docs describe a multimodal encoder that converts images/videos into a patch grid and merges them into the token sequence, with position handling done in a prefill step before text-only decoding. ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/qwen3_vl "Qwen3-VL - transformers."))

### 2.2 `prepare_inputs_for_generation` and dropping `pixel_values`

On the Hugging Face side, generation is driven by `prepare_inputs_for_generation`. For Qwen2-VL and Qwen2.5-VL, there is a critical snippet (paraphrased from the upstream and from a user patch):

- If `cache_position[0] != 0`, then **set `pixel_values = None` and `pixel_values_videos = None`**. ([Hugging Face](https://huggingface.co/MrLight/dse-qwen2-2b-mrl-v1/commit/adf727ac6accce2ed358280023cd49f291c98fbe "Update README.md · MrLight/dse-qwen2-2b-mrl-v1 at adf727a"))

This gate means:

- As soon as decoding starts (non-zero `cache_position`), the model is **forbidden from seeing images/videos** in `generate()`. They are forcibly removed from the inputs, exactly to enforce the “one multimodal prefill, then text only” pattern.

Qwen3-VL reuses the same conceptual pattern (prefill vs decode with RoPE computed once). The exact gating around pixel values may differ in details, but the key assumptions are the same:

- Images and videos are supposed to be processed in **prefill**.
- Decode with non-empty KV cache is assumed to be **text only**.

* * *

## 3. What this means for your two-image prefix-cache setup

Now apply the above to your code.

### 3.1 First call (prefix with `img1`)

- `past_key_values` is `None`, so the model is in **prefill** mode.

- All of the following happen:

This is the only point where the vision pathway and its RoPE indices are properly set up for that image.

### 3.2 Second call (new image `img2` + text, with `past_key_values`)

In the second call:

- You pass `past_key_values=prompt_cache` and a non-zero `cache_position`.

- `generate()` interprets this as **decode mode** :

In decode mode, for Qwen-VL-style models:

- Multimodal RoPE is **not recomputed** from `image_grid_thw` / `video_grid_thw`; it assumes images were already fully handled in prefill. ([Hugging Face](https://huggingface.co/jinaai/jina-embeddings-v4/blob/4a58ca57710c49f51896e4bc820e202fbf64904b/qwen2_5_vl.py "qwen2\_5\_vl.py · jinaai/jina-embeddings-v4 at ..."))
- In Qwen2-VL / Qwen2.5-VL, `prepare_inputs_for_generation` explicitly clears `pixel_values` when `cache_position[0] != 0`. ([Hugging Face](https://huggingface.co/MrLight/dse-qwen2-2b-mrl-v1/commit/adf727ac6accce2ed358280023cd49f291c98fbe "Update README.md · MrLight/dse-qwen2-2b-mrl-v1 at adf727a"))
- Qwen3-VL follows the same conceptual split and uses precomputed RoPE deltas in decode. ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/qwen3_vl "Qwen3-VL - transformers."))

So, depending on the exact version you run, one of two things happens:

1. **Strict behavior (like Qwen2-VL / Gemma-3)**

2. **Looser behavior but still prefill-only RoPE**

In either case, the model is not actually reasoning over a “fresh” `img2` alongside `img1`, so:

- When asked “Compare the images”, it either:

Exactly analogous behavior is documented for Gemma-3:

- In the Gemma-3 issue, `prepare_inputs_for_generation` is shown to forward `pixel_values` _only_ when `cache_position[0] == 0`, which makes prefix caching with a later image turn impossible. ([GitHub](https://github.com/huggingface/transformers/issues/40910 "Gemma-3: prepare\_inputs\_for\_generation should forward pixel\_values based on image token presence, not cache\_position==0 · Issue #40910 · huggingface/transformers · GitHub"))

So your Qwen3-VL and Gemma-3 experiments are hitting the same architectural assumption: **images are prefill-only**.

* * *

## 4. Evidence that this is a known limitation

Several public issues match your situation very closely:

1. **Prefix caching with Qwen3-VL & multimodal demos**  
A recent Reddit post describes a setup where each query has a fixed multimodal demo block (images + text) and a variable query, using Qwen3-VL with HF backend and manual prefix caching. It reports that behavior diverges once the prefix gets long but confirms that multimodal prefix caching is fragile and not officially supported. ([Reddit](https://www.reddit.com/r/learnmachinelearning/comments/1p33gau/prefix_caching_with_qwen3vl_fails_beyond_4096/ "Prefix Caching with Qwen3-VL Fails Beyond 4096 Tokens : r/learnmachinelearning"))

2. **Qwen2-VL / Qwen2.5-VL: images dropped when caching**  
A Hugging Face user patch explicitly shows that the upstream `modeling_qwen2_vl.py` sets `pixel_values` and `pixel_values_videos` to `None` whenever `cache_position[0] != 0`. ([Hugging Face](https://huggingface.co/MrLight/dse-qwen2-2b-mrl-v1/commit/adf727ac6accce2ed358280023cd49f291c98fbe "Update README.md · MrLight/dse-qwen2-2b-mrl-v1 at adf727a"))

3. **vLLM bug: `--enable-prefix-caching` + Qwen2-VL + second image**  
In vLLM issue #8296, the reporter states that enabling prefix caching for Qwen2-VL causes errors when the service processes a second image and notes that this parameter is currently incompatible with multimodal large models. ([GitHub](https://github.com/vllm-project/vllm/issues/8296 "[Bug]: Error when running multimodal large models with --enable-prefix-caching · Issue #8296 · vllm-project/vllm · GitHub"))

4. **Gemma-3: identical pattern**  
The Gemma-3 issue demonstrates that prefix caching with a later image turn is impossible because `prepare_inputs_for_generation` only forwards `pixel_values` at `cache_position[0] == 0`. After caching a text prefix, any subsequent turn that includes image tokens is processed without image features. ([GitHub](https://github.com/huggingface/transformers/issues/40910 "Gemma-3: prepare\_inputs\_for\_generation should forward pixel\_values based on image token presence, not cache\_position==0 · Issue #40910 · huggingface/transformers · GitHub"))

Taken together, these sources show that:

- Multimodal prefix caching is not just “unfinished”; the current implementations explicitly gate against it in the generation path.

* * *

## 5. What _does_ work today (recommended patterns)

Given the above, the following usage patterns are realistic and robust.

### 5.1 Put all images in a single prefill (no cross-call image caching)

For true visual comparisons, the only “clean” pattern right now is:

- Put **all images that need to be jointly reasoned about into a single prompt**.

- Call `generate()` **once** , so that:

- Do **not** try to reuse KV cache from a past call that already had images.

Conceptually:

```python
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": img1},
        {"type": "image", "image": img2},
        {"type": "text", "text": "Compare these two images."},
    ],
}]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512, use_cache=True)

```

This:

- Allows the model to actually see and compare two images.
- Avoids the decode-phase “no pixel\_values” restriction entirely, because images are only used in the prefill.

If deployment speed is the main concern:

- Use **engine-level prefix caching** that operates on the full tokenized prompt before it enters the model (e.g. vLLM’s internal prompt cache), once there is stable support for Qwen3-VL. But that must be a cache over the whole sequence including both images, not a KV reuse across different images.

### 5.2 Cache only pure text; always re-encode images

Another safe compromise:

- Restrict the KV cache to a **text-only prefix** :

- For every query:

This does not avoid the cost of the vision encoder for demo images, but it keeps caching semantics safe:

- Only the text is reused in `past_key_values`.
- Images and multimodal RoPE are recomputed per query as intended.

### 5.3 Summarize demo images to text once, then cache the text

If the main motivation is that the demo images are constant across queries, a pragmatic workaround is:

1. Run a one-time, “offline” multimodal call:

2. Take those descriptions and turn them into a **text-only prefix** :

3. Use that text-only prefix as the cached prompt for later comparisons:

Advantages:

- The KV cache is purely textual, which works reliably with current `past_key_values` semantics.
- The cost of encoding demo images is paid only once when generating the descriptions.
- Comparisons often work surprisingly well if the descriptions are detailed.

Trade-off:

- The model no longer has raw pixel access to both images simultaneously; it compares an image to a textual representation of another image. For many tasks this is acceptable; for fine-grained visual metrics it may be weaker.

### 5.4 If vLLM is used: avoid `--enable-prefix-caching` for multimodal

If using vLLM with Qwen-VL:

- The vLLM issue #8296 explicitly reports that enabling `--enable-prefix-caching` with Qwen2-VL and multiple images causes errors and is not compatible with multimodal models as currently implemented. ([GitHub](https://github.com/vllm-project/vllm/issues/8296 "[Bug]: Error when running multimodal large models with --enable-prefix-caching · Issue #8296 · vllm-project/vllm · GitHub"))

- Until there is an explicit fix in vLLM for Qwen3-VL multimodal caching, the safe assumption is:

Engine-side prompt caching may still work if it treats both text and image content as part of the cache key and rebuilds KVs per distinct multimodal prefix.

* * *

## 6. What would a “real” multimodal prefix cache require?

Conceptually, to truly support:

> “Cached multimodal prefix with image A, later reuse cache with new image B and text”

the model implementation would need:

1. **Non-destructive handling of `pixel_values` in decode**

2. **Recomputable multimodal RoPE state**

3. **A stable cache format that captures chunking / DCA metadata**

Implementing all this is non-trivial, and there is no official, production-quality patch yet for Qwen3-VL in the Hugging Face ecosystem.

* * *

## 7. Practical recommendation for your use case

Given the current state of Qwen3-VL and Gemma-3:

- The attempt in your snippet – reusing a multimodal KV cache and then adding a second image with a non-zero `cache_position` – is **not supported by the model’s generation code** and will remain unreliable.
- The failure mode (“the two images seem identical” or the second image is ignored) is expected.

For robust behavior now:

1. Use **single-prefill multimodal prompts** for any case where both images must be reasoned about together.

2. Restrict **KV prefix caching** (via `past_key_values`) to **text-only prefixes**.

3. If demo images must be reused:

* * *

### Summary

- Qwen3-VL (like Qwen2-VL / Qwen2.5-VL and Gemma-3) is designed for **“one multimodal prefill, then text-only decode”**.

- In the Hugging Face implementations, `prepare_inputs_for_generation` and the RoPE logic:

- As a result, the pattern “cached prefix with image 1 + new call with image 2 + text, using `past_key_values`” is not supported; the second image cannot be processed correctly.

- Similar problems are reported publicly:

- Recommended workarounds:

---

<div class="post-metadata">

**Author:** ![glasbruch](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/glasbruch/32/56035_2.png) [@glasbruch](https://discuss.huggingface.co/u/glasbruch)\
**Post date:** [November 26, 2025, 3:41pm UTC](https://discuss.huggingface.co/t/multimodal-prefix-caching-with-qwen3-vl/170849/3 "2025-11-26T15:41:04Z")

</div>

Thank you John, for the extremely detailed response and for summarizing the problem so clearly, it’s very helpful! I hope to find a safe way to implement this functionality soon. I attempted a simple local fix on my side (similar to the issue you mentioned above for Gemma), but recomputing the RoPE indices led to downstream errors that I haven’t yet been able to debug.