Anthropic’s models are improving twice as fast, mostly due to AI, METR report finds

The Verge reports on a METR report published on a Thursday after an 11-week investigation inside Anthropic, assessing the company's use of AI in its own research and development. METR found that Anthropic's models are improving about twice as fast as a year earlier, and benchmark scores point to about two years' worth of past progress compressed into the last twelve months. METR said it is confident the speed-up does not come merely from more training compute and that most of it as of October is due to AI.

Staff using the most capable internal model reported being around 12 times as productive as without AI, about double a figure given a few months before. Most of the gain currently comes from AI assisting human researchers, though METR sees signs that fully autonomous contributions will matter more soon, and it found no clear bottlenecks. Anthropic has also finished training three unreleased models, known to METR as Model 2, Model 3 and Model 4, finished training between July and October.

The report's limits drew attention. Anthropic could redact the public version, and METR said the company removed the majority of the evidence it found most informative from its roughly 100-page assessment. Anthropic cofounder Sam McCandlish said the redactions were commercially sensitive and could give others a roadmap, and he promised the company's own report in coming weeks. Critics, including Daniel Kokotajlo, accused Anthropic of hypocrisy, while Apollo Research's Marius Hobbhahn and the Institute for AI Policy and Strategy's Brianna Rosen called for more transparency and independent access. New York Assemblyman Alex Bores urged Congress to ban recursive self-improvement.