AgentLens Leaderboard
Coding agents on AgentLens-Bench, ranked by how well they solve tasks rather than how often. Quality is the mean AgentLens score of each model's passing trajectories.
| # | Model | Tier mix |
|---|
Click a row for its score distribution, tiers, and wasted steps.
Reading the table
- Quality
- Mean AgentLens quality score (0–100) over the model's passing trajectories, with a 95% confidence interval.
- Tier mix
- Share of passing trajectories that are Ideal, Solid, or Lucky. A Lucky Pass reaches a correct patch through a weak process.
- vs pass rate
- How many places the model moves when ranked by quality instead of pass rate.
How scores are computed
- For each task, AgentLens merges five passing runs of that task into a reference PTA, the space of known-good solutions. Every run is scored against a reference that does not contain it.
- A model's runs are scored only against runs from other models. Including its own runs would raise its score by about three points per same-model donor, an advantage a new agent never gets.
- The five donors are drawn at random. Every run is scored under five fixed draws, and the table averages them, because a single draw can shift scores by a few points.
- The donor lists for every draw, task, and model are published, and scoring needs no LLM calls, so every number on this page can be reproduced exactly.
Submit your agent
Run the benchmark with your own model and harness, score the runs locally, and send us the results.
- Download the repository and install the scorer. It contains the 47 tasks (
agentlens-bench/annotations/tasks.csv), the reference runs, and the scoring code. Scoring runs on CPU and needs no API keys.git clone https://github.com/agentlens-framework/agentlens cd agentlens && pip install ./sdk - Run your agent on each of the 47 SWE-bench Verified tasks, five times per task, and check every patch with the official SWE-bench harness. Any harness works if it saves OpenHands or ATIF trajectories.
- List your runs in a CSV file, one row per run: the task, the path to its trajectory file, and whether the harness marked it resolved.
task_id,trajectory,resolved django__django-11066,runs/django-11066/run1.json,true - Score the runs. This uses the same five reference draws as every model on this page and prints your quality score, pass rate, and Lucky rate.
python leaderboard/score_submission.py runs.csv --format openhands \ --name "My Agent" --model "my-model" --agent "MyHarness" \ --trajectories-url https://example.org/my-trajectories \ --out leaderboard/submissions/my-agent - Open a pull request that adds your
leaderboard/submissions/my-agentfolder. We re-score your trajectories from the link you provide and add your agent to this page.