Methods

The agentic axis is currently fed only by Terminal-Bench, which skews to software engineering and whose public leaderboard returned no rows this week — a real gap. AA-Briefcase, already folded into the Intelligence Index you track, is the closest available measure of agents doing non-coding professional work, so it directly fixes something the verdicts lack.

Fiction.liveBench, the only current source behind the long_context axis used by both the coding-agent and hard-problems verdicts, returned no usable benchmark table this run — that axis effectively has no working source right now. AA-LCR is a live eval already inside Artificial Analysis Intelligence Index v4.3.2 measuring the same thing, giving long_context a second, working source. Approving because it fills a real gap in what the verdicts can currently measure, not just because a new benchmark exists. No per-model AA-LCR scores were available this run, so there's nothing to apply yet beyond starting to track the method.

The maintainer of Aider (a popular AI coding tool) tests each model on 225 hard practice exercises across six programming languages. The model has to actually solve the problem and deliver the answer in exactly the format the tool requires — sloppy instruction- following costs points even when the underlying solution is right.

It complements SWE-bench nicely: SWE-bench is large messy real-world projects in one language; this is small hard puzzles across six languages. A model that scores well on both is genuinely good at code.

How to read a score: check the "edit format compliance" column alongside the solve rate. A model with a high solve rate but low compliance is like a brilliant employee who ignores instructions — the raw score flatters it.

Confirmed live in this run's fetch with a current value — Epoch's composite reads 167, topped by Claude Opus 5.5, across 87 benchmarks and 407 models. This matters now because Artificial Analysis version 4.3.2 has folded Terminal-Bench 4.0 into its own index, meaning the reasoning and agentic axes are no longer independent readings. Epoch is already a trusted source here and gives reasoning a second, separately-built instrument.

Every model advertises how much text it can take in at once — hundreds of pages, in some cases. This test checks whether the model actually understands text that long, by feeding it long serialized stories and asking comprehension questions that require holding the whole narrative in mind: who knows what, what happened before what. That's much harder than finding one planted sentence, which is what vendors' own demos usually show.

How to read a score: this site records the score at the longest length bucket (roughly a 500-page book). The shape of decline matters more than any single number — a model that scores 95% on short text and 60% at full length has a real working capacity far below what the label says. When two long-document tests disagree, trust neither and read both.

Aider's polyglot leaderboard is stale — no 2026-era model, newest entry is DeepSeek-V3.2-Exp from October 2025 — while FrontierCode 1.1 is already surfacing in the wild (cited for Claude Haiku 5.5) and comes from Artificial Analysis, the same independent measurement firm already trusted for the Intelligence Index and other tracked methods. Approving closes a real gap in how the coding axis gets measured.

The only agent measurement that returned real numbers this run — Claude Fable 5.1 at 14.31% and $5.43 a task, GPT-6.1 Sol at 11.23% and $0.56, with a cost-versus-quality frontier. Terminal-Bench, the existing agentic source, came back with an empty table, and nothing in the current method set pairs agent quality with price. Both the coding-agent and bulk-cheap verdicts weigh cost.

This run's SWE-bench fetch confirms Bash Only is now the default Verified view: the same 500 problems, every model run inside one shared agent environment instead of each lab's own scaffolding. Scaffold variance is the named honesty gap in the coding method we already track, so this changes how the coding axis is read, not just what feeds it.

The test takes 500 real bug reports filed against popular open-source software projects, each one checked by humans to make sure it's solvable. The model gets the bug report and the project's code, and has to produce a fix. The fix counts only if the project's own test suite passes — the same bar a human contributor's fix would have to clear. The score is simply the percentage of bugs fixed.

How to read a score: the supporting machinery matters almost as much as the model. A vendor's self-reported number using their own custom setup is not comparable to a stripped-down run, so prefer numbers the leaderboard maintainers verified themselves. Rule of thumb: a 5-point gap between comparable setups is a real difference in ability; a 2-point gap is coin-flip territory.

The test gives a model a working command-line environment and a goal — "this project won't compile, fix it," "clean up this dataset," "get this server configured" — then walks away. Success is checked mechanically: either the build passes, the data is right, the server runs, or it doesn't. No judges, no opinions. That makes it one of the cleanest measures of whether a model can do work rather than just answer questions.

How to read a score: every leaderboard entry names both a model and the harness it ran inside, and the harness matters a lot — same model, different harness, very different score. Comparing the same harness across models tells you about the models; comparing harnesses on one model tells you about the harnesses. This site records each model's best-harness number and marks who reported it.

Artificial Analysis is an independent measurement firm — think of them as a Consumer Reports for AI models. They run a fixed set of tests against each model's public service themselves, rather than trusting vendor claims, and combine the results into a single number. Because every model takes the same tests under the same conditions, comparing two models' index scores is meaningful in a way that comparing two vendors' press releases is not.

How to read a score: treat it as a ranking tool, not a grade. A 5-point gap means one model is clearly a tier above; a 1–2 point gap is within the noise of measurement. Check that two scores come from the same index version before comparing them — the recipe gets revised periodically.

The same firm also measures output speed, price, and performance on multi-step tasks; this site uses those measurements for the speed, cost, and agentic columns.

LMArena shows visitors two answers to the same question, hides which model wrote each, and asks "which is better?" Millions of these votes are turned into a rating using the same math as chess ratings: beat highly-rated opponents often and your rating climbs.

It's the best available measure of "which model do people like talking to" — which matters for a daily assistant — but likability and correctness are different things. A model can win votes by being confident and well-formatted while being subtly wrong.

How to read a score: gaps under about 15 points are a statistical tie. Use the "style-controlled" version of the ranking, which tries to correct for pretty formatting winning votes. And treat sudden ranking jumps right after a scoring-method change (like the August 2026 one) as an artifact of the change until they persist for a few weeks.