With an AI pair programmer, developers built the server 56% faster. Experienced maintainers on their own code were 19% slower — and felt 20% faster.

In May and June 2022, researchers at Microsoft Research and GitHub ran a controlled experiment on GitHub Copilot. They recruited 95 professional programmers through the freelancing platform Upwork and asked them to implement an HTTP server in JavaScript as quickly as possible; 45 were given Copilot and 50 were not. Among the 35 in each group who finished the task and the survey, the Copilot group took 71.17 minutes on average against 160.89 for the control group — 55.8% faster, with a 95% confidence interval of 21% to 89% and a p-value of 0.0017 [1].

The completion rate was seven percentage points higher with Copilot, a difference the authors report as not statistically significant. The gains were not evenly spread: developers with less programming experience and those who coded more hours a day benefited more, though the experience effect sat at the edge of significance (p = 0.063). The authors describe their own limits plainly — a standardized task rather than collaborative work on a large project, and no examination of code quality [1].

GitHub published the result as "55% faster" alongside a survey of more than 2,000 developers, which asked how using Copilot felt: between 60% and 75% said they were more fulfilled in their job, 73% that it kept them in the flow, and 87% that it preserved mental effort on repetitive tasks. The experiment measured minutes; the survey measured feelings [2].

Three years later, the research organisation METR ran a randomized controlled trial in the setting the first study had set aside. Sixteen experienced open-source developers, with an average of five years on their projects, worked 246 real issues from their own repositories — codebases averaging more than 22,000 GitHub stars and over a million lines of code — between February and June 2025, paid $150 an hour to be measured. Each issue was randomly assigned to allow or forbid AI tools; when allowed, the developers mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet, frontier models at the time [3][4].

With AI allowed, the developers took 19% longer to complete their issues. Before starting, they had forecast that AI would make them 24% faster. After finishing — having lived through the slowdown — they estimated it had made them 20% faster. Economists surveyed before the study had predicted 39% faster; machine-learning experts, 38% [4].

METR examined twenty properties of its setting that might explain the result and found evidence that five contributed. The developers were over-optimistic about the tools. They slowed down more on the issues they knew best, where their own expertise left the AI little to add. The repositories were large and complex. The AI's output was unreliable enough that developers accepted fewer than 44% of its generations, spent about 9% of their time reviewing and cleaning it on a 74-issue subset, and all of them reported having to modify what it produced, with 56% saying that often meant major changes. And the AI lacked the implicit knowledge of the codebase that its maintainers carried in their heads — it often did not pick the right place to make an edit [4].

METR's own report lists what the study does not show: that AI fails to speed up most developers, that the result generalizes beyond this setting or to other domains, or that future tools will not help here. Only 44% of its developers had used Cursor before, though 93% had used large language models and had substantial prior experience prompting them [3][4].

The Copilot gain is real in the shape it was measured: a well-specified, self-contained task, a control group and a stopwatch. And it landed where a pair programmer should — the developers with the least experience gained the most, which is why the authors describe the heterogeneity as promising for people entering the profession [1]. The industry-affiliated team also published the confidence interval, the p-value, the non-significant completion-rate result and their own limitations rather than only the headline [1]. Three years on, METR did the same in the other direction, printing the list of conclusions its result does not support [3]. Both trials are honest instruments; the disagreement between them is information about the setting, not a scandal.

Not the slowdown — the perception. Sixteen experienced developers, paid to be measured, with hundreds of hours of prompting behind them and the result freshly lived, judged that AI had made them 20% faster when it had made them 19% slower. The experts asked to forecast the outcome were further off still [4]. What the developers felt was real: with AI allowed they typed less and searched less, and the time moved to prompting, waiting on the model, reviewing its output and sitting idle — activities that feel like progress and do not register as effort [4]. GitHub's survey captured exactly this kind of feeling and reported it honestly as feeling [2]. The cost falls on whoever reads a feeling as a measurement: a team that mandates a tool because it feels faster, a maintainer who stops timing anything, a budget that assumes the 55% and inherits the 19%.