NeurIPS 2026

AgentLensRevealing The Lucky Pass Problem in SWE-Agent Evaluation

Priyam Sahoo1,2,*, Gaurav Mittal2,†, Xiaomin Li2,†, Shengjie Ma2, Benjamin Steenhoek2, Pingping Lin2, Yu Hu2

1University of Illinois Urbana-Champaign    2Microsoft

*Work done as a student researcher while at Microsoft.   †Co-second authors.

Histogram of quality scores for 1,136 passing trajectories, split into Lucky (122, 10.7%), Solid (785, 69.1%) and Ideal (229, 20.2%) bands.
Passing trajectories are not behaviorally homogeneous. Among 1,136 passing trajectories in AgentLens-Bench, AgentLens classifies 229 as Ideal (20.2%), 785 as Solid (69.1%), and 122 as Lucky (10.7%). Binary evaluation treats all of these trajectories as equally successful, while process-aware scoring separates direct, coherent solutions from weak processes that happen to pass.

Abstract

Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view treats a principled solution and a chaotic trial-and-error process as equivalent. We show that this equivalence is empirically false.

We evaluate 2,614 OpenHands trajectories from eight model backends on SWE-bench Verified. Of the 60 tasks in this corpus, 47 have enough passing trajectories to construct task-level process references, yielding a 1,815-trajectory evaluation subset. Among passing trajectories in this subset, 10.7% exhibit behavior we call a Lucky Pass: regression cycles, blind retries, missing verification, or temporally disordered exploration, implementation, and verification.

We introduce AgentLens, a framework for process-level assessment of SWE-agent trajectories, and define AgentLens-Bench, a dataset of 1,815 trajectories annotated with quality scores, waste signals, divergence points, and 47 task-level Prefix Tree Acceptor (PTA) references. AgentLens combines two components. First, it merges multiple passing solutions for the same task into a PTA reference space of correct behaviors. Second, it uses a context-sensitive intent-stage labeler that assigns actions to Exploration, Implementation, Verification, or Orchestration using trajectory history rather than tool identity alone.

On AgentLens-Bench, the composite score separates passing trajectories into Lucky, Solid, and Ideal tiers; decomposes Lucky Passes into five recurring mechanisms; and changes how the eight evaluated model backends are ranked compared with pass rate alone. Across these models, AgentLens classifies between 0.5% and 23.2% of successful trajectories as Lucky, and some models move by as many as five rank positions when ranked by quality score instead of pass rate.

How AgentLens works

AgentLens evaluates a candidate trajectory in four stages: it parses raw logs into labeled states, constructs a task-specific reference graph from passing solutions, scores the candidate against that reference, and returns a structured quality report.

Each action is assigned to one of four cognitive phases: Exploration Implementation Verification Orchestration, using trajectory history rather than tool identity alone. This resolves common ambiguities such as terminal commands: grep remains exploratory even after an edit, whereas pytest is verification.

AgentLens workflow in five panels: trace collection, trace parser and state creation, intent stage labelling, PTA construction and merging, and analysis and visualization.
AgentLens workflow. AgentLens starts from raw execution traces and converts them into structured state sequences. Intent-stage labeling assigns each state a process role. These intent-labeled states become the states used for per-trace PTA construction. Passing trajectories for the same task are then merged into a task-level PTA, which represents the known-good solution strategies for that task. New trajectories are scored against this task-level reference to produce quality tiers, divergence points, waste reports, and visual analyses.

A single reference trajectory cannot represent the diversity of correct strategies: two agents may solve the same task through different but valid sequences. AgentLens instead constructs a Prefix Tree Acceptor from multiple passing trajectories for the same task. Shared prefixes are merged into common nodes, while divergent but successful strategies form branches. Each root-to-terminal path therefore represents one known-good solution.

Two passing traces share a grep and read utils.py prefix, then diverge; the merged trace keeps the shared prefix and branches at the implementation step.
PTA construction. Two individual passing traces sharing an early exploration prefix but diverging at implementation are merged into a single DAG. Shared nodes reflect equivalent actions across agents; branches reflect genuine strategic divergence.

A candidate is scored against the task PTA with four signals: structural alignment, set coverage, trajectory coherence, and temporal profile similarity. They combine into a 0–100 quality score. Passing trajectories scoring 70 or more are Ideal, 47 to 70 are Solid, and below 47 are Lucky.

Results

Lucky Passes reveal recurring weak-success mechanisms

The 122 Lucky Passes produce correct patches but receive low process-quality scores. The two largest categories are Brute-Force Convergence, where the agent reaches a correct patch through repeated low-coherence attempts, and Incomplete Implementation, where the agent makes a partial fix that passes because visible tests are insufficient. Lucky trajectories with blind retries waste 11.4 steps per instance, compared with 2.7 in Ideal trajectories.

Lucky Pass taxonomy. C2 and C3 together account for 68.0% of all Lucky Passes.
Categoryn%Key signalPrimary cause
C1: Minimal & Unverified1915.6Short + no V stageAgent overconfidence
C2: Brute-Force Convergence4234.4High waste, low coh.Lack of planning
C3: Incomplete Implementation4133.6Partial fix, low cov.Test-suite gaps
C4: Excessive Exploration54.1Very long, unfocusedMissing termination
C5: Divergent-but-Valid1512.3Alternative approachMultiple valid solutions

Process quality changes how models are ranked

Quality-score rankings disagree with pass-rate rankings for every model. GPT-4o rises from 8th by pass rate to 3rd by quality score among the tasks it solves. Opus 4.6 shows the opposite pattern: a 77.3% pass rate hides an 18.7% Lucky rate.

Frontier model comparison. Quality-score rankings (QS rank) disagree with pass-rate rankings (PR rank) on all eight models.
ModelPass %PR rankQualityQS rankLucky %
sonnet-4.586.8267.411.0
opus-4.587.9166.220.5
gpt-4o34.9863.434.1
gemini-2.5-pro42.9759.247.6
gpt-5.3-codex45.9658.3515.3
opus-4.677.3356.7618.7
gpt-5.2-codex64.6456.1719.4
gpt-4.159.9554.7823.2

AgentLens-Bench

AgentLens-Bench is a process-annotated SWE-agent trajectory dataset built from 47 PTA-eligible SWE-bench Verified tasks. Each trajectory has a 40-column feature vector with quality scores, waste annotations, divergence metadata, and a tier label, and each task has one ground-truth PTA. Because each PTA represents the known-good solution space for a distinct task, the collection provides task-diverse references for trajectory scoring, filtering, and failure analysis.

See the leaderboard · Browse the dataset on GitHub (CC-BY-4.0)

BibTeX

@inproceedings{sahoo2026agentlens,
  title     = {AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation},
  author    = {Sahoo, Priyam and Mittal, Gaurav and Li, Xiaomin and Ma, Shengjie and Steenhoek, Benjamin and Lin, Pingping and Hu, Yu},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2605.12925}
}