Models

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

Gives the model real bug reports from actual software projects and checks whether its fix passes the project's own tests — the closest thing to watching it do a programmer's job.

Drops the model into a computer's command line with a task like "get this software building again" and checks whether the end result actually works — a test of getting real multi-step work done, not answering questions.

Tests whether a model that claims to handle book-length input actually keeps the story straight all the way through, or quietly loses the plot — advertised capacity and real comprehension often differ a lot.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

Gives the model real bug reports from actual software projects and checks whether its fix passes the project's own tests — the closest thing to watching it do a programmer's job.

Drops the model into a computer's command line with a task like "get this software building again" and checks whether the end result actually works — a test of getting real multi-step work done, not answering questions.

Tests whether a model that claims to handle book-length input actually keeps the story straight all the way through, or quietly loses the plot — advertised capacity and real comprehension often differ a lot.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

Gives the model real bug reports from actual software projects and checks whether its fix passes the project's own tests — the closest thing to watching it do a programmer's job.

Tests whether a model that claims to handle book-length input actually keeps the story straight all the way through, or quietly loses the plot — advertised capacity and real comprehension often differ a lot.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

Gives the model real bug reports from actual software projects and checks whether its fix passes the project's own tests — the closest thing to watching it do a programmer's job.

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

Gives the model real bug reports from actual software projects and checks whether its fix passes the project's own tests — the closest thing to watching it do a programmer's job.

An independent firm runs the same battery of tests on every model and blends the results into one overall smarts score — useful for comparing models against each other, not as an absolute measure.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.

Thousands of people vote blind on which of two anonymous answers they prefer, producing a popularity rating — it tells you which answers people like, which is not the same as which are right.