AI Is Getting Too Good Too Fast—And Researchers Are Running Out of Ways to Measure It

For months, a single graph has been circulating through Silicon Valley’s group chats and boardrooms with the kind of reverence usually reserved for stock charts or epidemiological curves. Created by the nonprofit research institute METR, it tracks something deceptively simple: how long of a software task AI models can complete successfully. But the curve on that graph tells a story that has even the most measured researchers using words like “insane” and “chaotic.”

According to METR’s latest analysis, artificial intelligence is getting twice as capable roughly every seven months. The trend isn’t just continuing—it’s accelerating. And the most recent data point, representing Anthropic’s Claude Opus 4.6, has sent the already-feverish conversation into overdrive.

“I feel very confident now that it’s going to be totally insane and chaotic, like many orders of magnitude more chaotic than anything the world has experienced in our lifetimes.” — METR researcher, in a note to college friends

The Measurement Problem

METR, short for Model Evaluation and Threat Research, has become an unlikely focal point for the AI industry. The organization tests AI systems by assessing their ability to complete increasingly complex human software tasks. It’s straightforward in theory: give an AI a coding task, measure how long it takes, and track the trend over time.

The exponential curve that has emerged from this work has drawn comparisons to the early days of the COVID-19 pandemic—another situation where exponential growth transformed seemingly small increases into monstrous leaps. As one UK tech entrepreneur put it: “Nothing, nothing, nothing, everything.”

But here’s where the story gets complicated. METR’s own researchers are increasingly nervous about the measurements they’re publishing. The confidence intervals on their latest Claude Opus 4.6 evaluation are extremely wide, reflecting genuine uncertainty about exactly how capable these systems have become.

The testing bottleneck has become a paradox of its own. METR is running out of tasks hard enough to properly test the latest AI models. When your evaluation framework can’t find challenges difficult enough to stress-test the technology, that itself becomes data—just not the kind that’s easy to chart.

“We’re increasingly nervous about the measurements that we’re putting out there. We don’t want to hide behind that. I think that’s real uncertainty.” — Joel Becker, METR technical staff

What the Numbers Actually Mean

For all the excitement around the METR chart, the details matter. The graph measures task length that AI can complete 50% of the time—meaning failure is still the norm at the edge of capability. Even at 80% success rates, we’re nowhere near the kind of reliability that would enable full automation in corporate environments.

Enterprise reality hasn’t caught up to the benchmarks yet. A business that turned its operations over to an AI succeeding half the time wouldn’t survive the quarter. The gap between benchmark performance and production deployment remains significant, even as that gap narrows with each new model release.

The self-improvement question looms over everything. METR’s Joel Becker, who told Sky News he stopped paying into his pension after understanding the trajectory of AI development, believes we’re not yet at the point where AI can meaningfully improve itself—the recursive scenario that fuels science fiction fears of runaway capability explosions.

But Becker also notes something nearly as significant: AI tools are already accelerating the pace at which AI professionals can build better systems. The feedback loop may not be fully closed, but it’s tightening.

The Employment Paradox

For all the anxiety about AI replacing workers, the employment data tells a more nuanced story. Software engineering job postings on Indeed are actually rising, not falling. The UK and US labor markets show little sign of AI-driven disruption—at least not yet.

The lag effect is critical here. Economic statistics reflect what happened months ago, not what’s happening today. The extraordinary progress in software engineering capabilities has occurred largely in the past few months—too recent to show up in employment data.

Becker expects coders to have a future “for a while at least.” The AI professionals inside labs are still doing real work, and that work isn’t disappearing overnight. But the definition of that work is shifting rapidly, and the skills that matter are evolving just as fast.

The Industry Response

The METR chart hasn’t just captured attention—it’s shaping behavior. Markets move on small changes in AI assessments. Investment decisions are being made based on trajectories that may or may not hold. The pressure to deploy is building against the uncertainty of what these systems can actually do reliably.

Demis Hassabis, the famously measured CEO of Google DeepMind, regularly states that AI will have ten times the impact of the Industrial Revolution in one-tenth of the time. Even the most cautious leaders in the field are using language that would have seemed hyperbolic just a few years ago.

The question isn’t whether AI is advancing rapidly. The question is whether our frameworks for understanding that advance are keeping pace. When the measurement tools start breaking down, when researchers admit they’re not sure how to test the systems they’re evaluating, we’re in uncharted territory.

“I want to communicate that the situation is serious, that it’s fast-moving, that it appears not to be slowing down, that it is accelerating. It could be associated with extraordinarily positive possibilities… and on the other side, there may be extraordinary, dangerous things that might follow.” — Joel Becker, METR

The Road Ahead

The next few months will test whether the exponential trend holds. New models from OpenAI, Google, Anthropic, and others will either confirm the trajectory or introduce new variables that complicate the picture. The measurement problem may get worse before it gets better.

For now, the industry is operating on a mix of hard data and informed intuition. The METR chart provides a reference point, but everyone watching it knows the error bars are growing along with the capabilities. We’re measuring something that may be becoming immeasurable.

What comes next depends on whether the trend continues, and whether our institutions can adapt to a pace of change that has already outstripped many of our frameworks for understanding it. The graph that has everyone talking may soon be obsolete—not because it’s wrong, but because it’s no longer sufficient.

This article was reported by the ArtificialDaily editorial team. For more information, visit Sky News and METR.