Is 3B Always Better Than 1.5B? Why Qwen2.5-1.5B Beat 3B in Our Long-Context Benchmark

Research
1

Small Language Models (SLMs) are increasingly deployed on edge devices and local hardware. While modern architectures boast theoretical context windows of up to 128K tokens, real-world resource constraints often force users to operate under significantly smaller context loads. More importantly, standard long-context benchmarks heavily rely on retrieval-oriented tasks (such as Needle-in-a-Haystack or single-fact QA).

However, a fundamental research question remains: Does a model’s ability to retrieve a single isolated fact mean it can preserve and reconstruct an entire context of accumulated information?

To investigate this gap, I developed and evaluated SRRM (Simple Retrieval-Reconstruction Memory), a lightweight and reproducible benchmark designed to measure long-context information retention in SLMs from two complementary angles.

SRRM jointly evaluates models across progressively increasing context loads (from 10 to 100 factual statements):

  1. Targeted Retrieval (Sequential Retrieval Test): Tests the model’s ability to recall specific, isolated facts through explicit direct questions.

  2. Global Information Reconstruction (Summary Reconstruction Test): Prompts the model to reproduce all previously presented information in a single structured list without individual cues.

image
image563×766 62.3 KB

image
image942×455 58.3 KB

image
image704×434 36.9 KB

To link these two capabilities, SRRM introduces the Memory Preservation Ratio (MPR):

\text{MPR} = \frac{\text{Mean CIR Score}}{\text{Mean Retrieval Accuracy}}

This metric quantifies how much of the successfully retrieved information actually survives when the model is asked to perform holistic context reconstruction.

Key Findings from Evaluating Qwen2.5 (0.5B, 1.5B, and 3B)

Using a standardized synthetic dataset interweaving four semantic categories (hair color, family relations, activities, and WiFi passwords), we evaluated three parameter sizes of the Qwen2.5 family via Ollama under identical hardware and context limits (4096 tokens).

1. Retrieval Accuracy vs. Reconstruction Degradation

Across all models, global reconstruction performance (Critical Information Recall / CIR) degraded significantly faster than targeted retrieval accuracy as context size grew.

  • For instance, at 100 facts, Qwen2.5-1.5B maintained a 75.0% retrieval accuracy, but its reconstruction recall dropped to 29.0%.

  • This confirms that targeted retrieval alone drastically overestimates a model’s true long-context memory preservation.

2. Non-Linear Scaling Behavior

Surprisingly, increasing model parameter size did not translate to better long-context memory retention:

  • Qwen2.5-1.5B emerged as the top performer across all benchmark indicators, achieving an average retrieval accuracy of 86.7%, a reconstruction score of 45.7%, and an MPR of 0.53.

  • Qwen2.5-3B performed noticeably worse, recording 72.2% retrieval accuracy, a low 18.5% reconstruction recall, and an MPR of just 0.26.

  • The smaller Qwen2.5-0.5B achieved an intermediate MPR of 0.46.

This suggests that context retention efficiency and internal compression behavior in SLMs are heavily influenced by architecture/optimization dynamics beyond pure parameter count.

srrm-graph (1)
srrm-graph (1)1881×1063 79.7 KB

Model Mean Ret. Mean CIR MPR Interpretation
Qwen2.5-0.5B 65,1% 29,7% 0,46 Moderate
Qwen2.5-1.5B 86,7% 45,7% 0,53 High
Qwen2.5-3B 72,2% 18,5% 0,26 Low

:magnifying_glass_tilted_left: Why This Matters for SLM Deployment

In real-world multi-turn applications, such as personal AI assistants, local agentic workflows, and document summarization, models aren’t just asked isolated queries; they need to synthesize and maintain holistic awareness of prior conversation history.

Evaluating models solely with retrieval accuracy hides critical failure modes during long-form generation. Benchmarks like SRRM help surface these complementary memory behaviors.

Paper & Code Access

I’d love to hear the community’s feedback, thoughts on long-context compression mechanisms, or ideas on extending SRRM to multimodal or larger models!

2

nice idea, retrieval vs reconstruction is a real gap and most long-context benchmarks only test the first one.

before reading the 1.5B vs 3B result as a scaling effect, i’d rule out two boring explanations. first, length: at 100 facts the reconstruction answer is a long list, and with a 4096 context the prompt plus that list can run out of room. ollama returns done_reason, so check how many reconstruction runs ended on “length”. a model that writes slightly longer lines gets cut off earlier and loses recall for reasons that have nothing to do with memory.

second, format. if the scorer parses a structured list, a model that numbers items differently or merges two facts on one line scores low even when it remembers them. i ran into exactly this in a small study of mine: a strict scorer made one setup look clearly worse, and on the final answers the difference wasn’t significant. reading 10 of the 3B’s worst reconstructions by hand usually settles which one it is.

and i’d run each condition a few times with different seeds and fact orders and report intervals, since one synthetic set per load level can swing a lot.

3

Thanks a lot for this incredibly thoughtful and rigorous feedback! This is exactly the kind of valuable critique I was hoping to get from the community.

To address the points you raised based on how the benchmark was executed:

  1. Format & Scorer Artifacts: I actually ran manual human evaluations and live output monitoring throughout the runs specifically to catch this. The scoring logic handles minor formatting shifts, and during manual checks on the 3B model’s lower-scoring runs, the drop was genuinely driven by omitted or forgotten facts rather than parser misinterpretations or line-merging artifacts.
  2. Dataset Variance & Seeds: Thank you for bringing up confidence intervals! I previously tested across multiple random seeds and shuffled fact orderings, where the non-linear trend consistently held, but I hadn’t formally reported confidence intervals in the write-up. Pointing that out was super helpful, and I’ll definitely include CI metrics in the next revision to make the statistical backing clearer.
  3. Generation Length Limits (done_reason): This is a great point. While the context limit was set to 4096 tokens, explicitly stress-testing num_ctx variations and tracking done_reason for generation cutoffs is something I’m eager to explore in follow-up experiments as a core variable.

Really appreciate you sharing your experience from your own study, this helps refine the benchmark moving forward!