Methods
Every source this site cites, in plain language: what a score there actually tells you, what it misses, and how much to trust it. Expand any entry — no need to leave the page.
Every source has a lens
No outlet is neutral — labs market themselves, critics assume the worst, consultancies sell the cure. This site's discipline is to name each lens and attribute opinions to their holders rather than absorb them as fact. The characterizations below are the owner's judgments; the pipeline reads them on every run.
All sources and their lenses
primary
- Anthropic — News — First-party marketing. Factual on specs and pricing; promotional on quality and safety claims.
- Anthropic — API changelog — First-party, factual — changelogs rarely spin, but they also omit.
- OpenAI — News — First-party marketing; treat capability and mission claims as claims.
- OpenAI — API changelog — First-party, factual with omissions.
- Google DeepMind — Blog — First-party; research-prestige framing, consumer-first priorities show in what gets announced.
- Google — Gemini API changelog — First-party, factual with omissions.
- Meta — Newsroom (AI) — Corporate newsroom — the most PR-shaped of the primary feeds; verify against independents.
- Mistral — News — First-party; sovereignty/open-weights positioning is part of its sales narrative.
- DeepSeek — News — First-party; terse and factual, but benchmark claims are lab-reported until reproduced.
- Qwen — Blog — First-party; release-cadence marketing, benchmark claims lab-reported.
- xAI — News — First-party; superlative-heavy announcements, verify capability claims independently.
- Cohere — Blog — First-party; enterprise positioning.
- AI21 — Blog — First-party.
aggregator
- Artificial Analysis — Intelligence Index — Independent measurement firm — the closest to neutral here; its lens is which tests it chooses to blend.
- Artificial Analysis — Output speed — Independent measurement; methodology is the lens.
- Artificial Analysis — Pricing — Independent; list prices only, not negotiated reality.
- Artificial Analysis — Agentic index — Independent; composite construction is a judgment.
- Epoch AI — Benchmarks — Research institution; careful, slow, forecast-oriented.
leaderboard
- LMArena — Text leaderboard — Crowd preference, not correctness — measures what voters like.
- LMArena — WebDev leaderboard — Crowd preference within one task type.
- Hugging Face Hub — Trending models — Popularity signal — downloads measure attention and accessibility, not quality.
benchmark
- SWE-bench Verified leaderboard — Academic benchmark; scaffold variance is the honesty gap.
- Aider polyglot leaderboard — One respected practitioner's harness; rigorous but singular.
- Terminal-Bench leaderboard — Academic, outcome-verified; task selection skews software engineering.
- tau-bench results — Deprecated-era benchmark — treat current entries skeptically.
- Fiction.liveBench long-context — Niche community benchmark; narrow domain, honest method.
curator
- Simon Willison — Enthusiast-practitioner; tool-positive, hype-averse on claims he can't reproduce.
- Nathan Lambert — Interconnects — Researcher lens; open-model-sympathetic, strong on lab strategy.
- Latent Space — Builder/investor-adjacent; adoption-optimistic.
- Zvi Mowshowitz — Don't Worry About the Vase — Rationalist risk lens; deeply skeptical of lab safety claims and incentives.
- Import AI — Policy/research lens.
- Platformer (Casey Newton) — Power-and-accountability lens; skeptical of platform owners, strong on the people/governance story.
- Ed Zitron — Where's Your Ed At — Sharply critical of AI-industry economics — assume a negative lean; his factual digging is often solid, his conclusions are opinion.
- Stratechery (Ben Thompson) — free articles — Business-strategy lens (aggregation theory); market-sympathetic, treats profit motive as explanatory rather than indicting.
- MIT Sloan Management Review — Management-research orthodoxy; evidence-based, slow. (Shutting down Sept 2026.)
- Knowledge at Wharton — Academic management lens; institutionally cautious.
- McKinsey Insights — Sells transformation consulting — adoption optimism is a commercial interest; use for what enterprises are being told, not for ground truth.
- The AI Daily Brief (podcast transcript) — Practitioner news podcast; adoption-positive.
press
- TechCrunch — AI — Startup/VC-ecosystem-friendly; funding numbers reliable, framing boosterish.
- Crunchbase News — Funding data with light narrative; numbers good, significance framing thin.
- CIO Dive — Enterprise-IT practitioner trade press; vendor-neutral-ish, survey-heavy.
- The Register — AI/ML — Sardonic, vendor-critical; good at deployment-failure stories others skip.
- 404 Media — Adversarial accountability journalism; assumes power will misbehave, then documents it.
owner
- Owner notes — The owner's own measurements and judgments, labeled as such.
Benchmarks — structured tests with checkable answers
Aider polyglot leaderboard medium trust feeds: coding expand ▾ One developer's rigorous test of models solving hard programming exercises in six languages — a useful second opinion alongside SWE-bench, run the same way for every model.
- Tells you
- Whether a model can solve hard, self-contained programming exercises in six different programming languages, and follow precise editing instructions while doing it.
- Does not tell you
- Finding its way around a large existing project, real bug reports, speed, cost.
- Watch out for
- The exercises are public, so models may have seen them during training; the whole benchmark is maintained by one person; results depend on his specific tool.
- Run by
- Paul Gauthier (Aider)
- Last reviewed
- 2026-08-01
The maintainer of Aider (a popular AI coding tool) tests each model on 225 hard practice exercises across six programming languages. The model has to actually solve the problem and deliver the answer in exactly the format the tool requires — sloppy instruction- following costs points even when the underlying solution is right.
It complements SWE-bench nicely: SWE-bench is large messy real-world projects in one language; this is small hard puzzles across six languages. A model that scores well on both is genuinely good at code.
How to read a score: check the "edit format compliance" column alongside the solve rate. A model with a high solve rate but low compliance is like a brilliant employee who ignores instructions — the raw score flatters it.
Fiction.liveBench medium trust feeds: long_context expand ▾ Tests whether a model that claims to handle book-length input actually keeps the story straight all the way through, or quietly loses the plot — advertised capacity and real comprehension often differ a lot.
- Tells you
- Whether a model actually understands a very long document end to end — who knew what, when things happened — rather than just claiming a big capacity on the label.
- Does not tell you
- Long business documents or code specifically (it uses fiction), finding a single planted fact, speed on long inputs.
- Watch out for
- Built on fan fiction, which is a narrow style of text; question quality varies; relatively few questions at each length.
- Run by
- Fiction.live
- Last reviewed
- 2026-08-01
Every model advertises how much text it can take in at once — hundreds of pages, in some cases. This test checks whether the model actually understands text that long, by feeding it long serialized stories and asking comprehension questions that require holding the whole narrative in mind: who knows what, what happened before what. That's much harder than finding one planted sentence, which is what vendors' own demos usually show.
How to read a score: this site records the score at the longest length bucket (roughly a 500-page book). The shape of decline matters more than any single number — a model that scores 95% on short text and 60% at full length has a real working capacity far below what the label says. When two long-document tests disagree, trust neither and read both.
SWE-bench Verified high trust feeds: coding expand ▾ Gives the model real bug reports from actual software projects and checks whether its fix passes the project's own tests — the closest thing to watching it do a programmer's job.
- Tells you
- Whether a model can fix real reported bugs in established software projects, working the way a human programmer would — read the bug report, find the problem in the code, write a fix that passes the project's tests.
- Does not tell you
- Building something new from scratch, visual/front-end work, languages other than Python, how fast or cheap the model is.
- Watch out for
- Vendors test with different amounts of supporting machinery around the model, so their numbers aren't directly comparable; older test items may have leaked into training data, letting models "remember" answers.
- Run by
- OpenAI / Princeton
- Last reviewed
- 2026-08-01
The test takes 500 real bug reports filed against popular open-source software projects, each one checked by humans to make sure it's solvable. The model gets the bug report and the project's code, and has to produce a fix. The fix counts only if the project's own test suite passes — the same bar a human contributor's fix would have to clear. The score is simply the percentage of bugs fixed.
How to read a score: the supporting machinery matters almost as much as the model. A vendor's self-reported number using their own custom setup is not comparable to a stripped-down run, so prefer numbers the leaderboard maintainers verified themselves. Rule of thumb: a 5-point gap between comparable setups is a real difference in ability; a 2-point gap is coin-flip territory.
Terminal-Bench high trust feeds: agentic expand ▾ Drops the model into a computer's command line with a task like "get this software building again" and checks whether the end result actually works — a test of getting real multi-step work done, not answering questions.
- Tells you
- Whether a model can complete real multi-step technical tasks on its own — fixing a broken software build, wrangling data, administering a system — where success is checked by whether the end result works, not by opinion.
- Does not tell you
- Anything involving a visual interface or web browsing, working with people, tasks outside software and systems work.
- Watch out for
- Results depend heavily on the harness wrapped around the model (the scaffold), so the same model can post very different scores; tasks lean toward software engineering.
- Run by
- Stanford / Laude Institute
- Last reviewed
- 2026-08-01
The test gives a model a working command-line environment and a goal — "this project won't compile, fix it," "clean up this dataset," "get this server configured" — then walks away. Success is checked mechanically: either the build passes, the data is right, the server runs, or it doesn't. No judges, no opinions. That makes it one of the cleanest measures of whether a model can do work rather than just answer questions.
How to read a score: every leaderboard entry names both a model and the harness it ran inside, and the harness matters a lot — same model, different harness, very different score. Comparing the same harness across models tells you about the models; comparing harnesses on one model tells you about the harnesses. This site records each model's best-harness number and marks who reported it.
Aggregators — independent firms measuring everything the same way
Artificial Analysis Intelligence Index high trust feeds: reasoning expand ▾ An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.
- Tells you
- General problem-solving ability — a blend of reasoning, knowledge, math, and coding tests, all run by the same independent team on each model's public service, so every model faces identical conditions.
- Does not tell you
- Long multi-step projects, taste and judgment in writing, performance on very long documents, anything requiring the model to use tools.
- Watch out for
- The recipe for blending the tests changes over time, so scores from different index versions aren't directly comparable.
- Run by
- Artificial Analysis
- Last reviewed
- 2026-08-01
Artificial Analysis is an independent measurement firm — think of them as a Consumer Reports for AI models. They run a fixed set of tests against each model's public service themselves, rather than trusting vendor claims, and combine the results into a single number. Because every model takes the same tests under the same conditions, comparing two models' index scores is meaningful in a way that comparing two vendors' press releases is not.
How to read a score: treat it as a ranking tool, not a grade. A 5-point gap means one model is clearly a tier above; a 1–2 point gap is within the noise of measurement. Check that two scores come from the same index version before comparing them — the recipe gets revised periodically.
The same firm also measures output speed, price, and performance on multi-step tasks; this site uses those measurements for the speed, cost, and agentic columns.
Leaderboards — live rankings from votes or submissions
LMArena (text) medium trust feeds: human_preference expand ▾ Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.
- Tells you
- Which model's answers ordinary people prefer, when shown two anonymous responses side by side and asked to pick — scored like a chess rating, where beating strong opponents raises your number.
- Does not tell you
- Whether answers are actually correct on hard problems, performance on long projects, anything a voter can't judge in a quick read.
- Watch out for
- Long, confident, nicely formatted answers win votes even when they're wrong; vendors quietly test unreleased models here to tune for votes; the scoring method was tightened in Aug 2026, shuffling rankings.
- Run by
- LMArena (formerly LMSYS)
- Last reviewed
- 2026-08-16
LMArena shows visitors two answers to the same question, hides which model wrote each, and asks "which is better?" Millions of these votes are turned into a rating using the same math as chess ratings: beat highly-rated opponents often and your rating climbs.
It's the best available measure of "which model do people like talking to" — which matters for a daily assistant — but likability and correctness are different things. A model can win votes by being confident and well-formatted while being subtly wrong.
How to read a score: gaps under about 15 points are a statistical tie. Use the "style-controlled" version of the ranking, which tries to correct for pretty formatting winning votes. And treat sudden ranking jumps right after a scoring-method change (like the August 2026 one) as an artifact of the change until they persist for a few weeks.