The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements
Abstract
Agent evolution is often bottlenecked by noisy evaluations that cannot reliably identify better candidate artifacts or rank methods, so progress requires making true improvements detectable and accumulable.
Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher. This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.
Community
TL;DR: Much of the current community effort on agent evolution / RSI (across skills, harnesses, memory, etc.) is focusing on the wrong bottleneck. Proposing better artifacts is actually quite easy—the real challenge is reliably identifying whether an artifact truly helps. In fact, many seemingly impressive gains reported in recent work come down to exploiting evaluation shortcuts.
Everything you want to know about the reality of agent evolution is in this paper. For instance, we mapped out the actual score density distributions across mainstream evolution methods, frontier models, and standard benchmarks (spanning 1,030+ evolution runs and 10M+ LLM calls!). Everyone in the field has wanted to see these curves, though computing each one is prohibitively expensive—so we bit the bullet and absorbed the cost to save the community from hitting the same walls.
Surprise: Many of these score distributions aren't Gaussian at all—they are multimodal!
We also packed the paper with tons of hard-earned, practical insights:
- How can a single sentence hack mainstream benchmarks and reproduce 90% of full evolution gains?
- Are there actually transferable skills that genuinely boost performance on Terminal-Bench v2.1? (Spoiler: Yes, and we show why standard selectors miss them.)
Plus plenty more actionable know-how inside. If you are building or evaluating self-evolving agents, you'll definitely walk away with something valuable!
🔗 Project Page: https://lineshogan.github.io/resolution-hypothesis/
The finding that typical method margins are smaller than the gap between two runs of the same method applies well beyond evolution. It's the same trap when teams compare their current model against a candidate on a small eval set and ship on a 2-point difference. Did paired, per-case comparison (same items, counting wins vs losses) separate helpful from harmful candidates better than comparing mean scores? I'm curious whether the "no better than random" result at the candidate level holds under that kind of selection too.
Fair point on small eval sets—though to be fair to frontier model labs, while public benchmarks can be noisy, as far as I know they usually rely on massive internal eval suites to reliably spot real model gains. Evolving agents have it much tougher since they usually only get a relatively small test budget at each step.
On paired per-case wins vs. losses: on pass/fail tasks over the same test set, counting wins minus losses actually picks the exact same winner as comparing average scores (ties just cancel out), so single-run selection still sits at 49.7%—basically a coin flip. More importantly, averaging across many tasks actually shrinks overall variance. If you don't average and instead look at single tasks, per-task variance is huge (one random hiccup flips a pass to a fail), making candidate differences just as hard to tell apart.
If you haven't yet, definitely try the mini-game and interactive demos on our project page (https://lineshogan.github.io/resolution-hypothesis/)—they give a really fun, intuitive feel for how this plays out!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Does Your Generalist Skill Optimizer Really Incentivize Agentic Tasks? Rethinking Skill Optimization via Bipartite Attribution (2026)
- SkillFocus: Evolving Agent Skills via Capability Decomposition (2026)
- VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks (2026)
- Beneath the Surface: What We Overlook When We Talk About Agent Evolution (2026)
- SkillNEO: Non-Elitist Evolutionary Optimization of Agent Skills (2026)
- SAGE: A Statistical Acceptance Gate for Self-Evolving Agents (2026)
- EvoSkill v2: Fully Agentic Skill Evolution from Benchmark Adaptation to a Stateful Evolver (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Thanks for locating the bottleneck in identifying improvements rather than proposing them, and for reporting that candidate-level selection can be no better than random even when helpful artifacts exist. The method-level finding that typical margins are smaller than the difference between two runs of the same method is a useful caution (I have not rerun the experiments).
A question I would like to understand better: once a noisy evaluation accepts a candidate artifact (a prompt, skill, or memory), can a later round retract or refuse to keep it if a subsequent evaluation disagrees, or does an accepted artifact tend to remain in the pool? I am trying to distinguish improving the accept-time identifier from an explicit rollback or non-adoption rule after a noisy win.
The transferable skills that standard selectors missed seem related: if a different identification procedure recovered them, did that procedure also reduce the accumulation of harmful acceptances?
— Jack Tan (@Jacktan007 )
In most current setups, accepted artifacts stay in the pool, so lucky flukes pile up—for instance, we saw runs that looked +9 points better during evolution actually drop -3 points when we re-tested them 8 times to average out luck.
Backtesting accepted artifacts later to roll back bad ones makes total sense, and we actually tried this outside the paper, but it's really hard in practice. First, you need a reason to suspect something broke before you even start looking for a bad artifact—but noise is so high that you can barely tell a real regression from an unlucky run. Second, pinning down the culprit basically comes down to two approaches: statistical ones (like backtesting, which hits the exact same noise problem) or prior-based ones (like LLM judges or trendy gradient-based attribution, which turned out pretty unreliable in practice—even when just picking the better of two candidates, LLM judges got 45%–49%, no better than guessing).
And yes to your last question! Getting a cleaner signal before accepting (like averaging more runs) both uncovers real transferable skills and keeps harmful ones out upfront. Also, feel free to play around with the mini-game and interactive visuals on our project page (https://lineshogan.github.io/resolution-hypothesis/)—they make these dynamics super intuitive to see firsthand!
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper