How To Measure If AI Agents Actually Improve Developer Productivity (opens in new tab)

In 2025, a research nonprofit called METR ran a careful experiment. They took 16 experienced open-source developers, gave them 246 real tasks on codebases they'd worked in for years, and randomly let them use AI tools on some tasks and not others. Then they timed everything. The developers expected AI to make them about 24% faster. After the study, they reported feeling about 20% faster. They were actually 19% slower. Read that again, because it's the whole problem in three numbers. The peopl...