AI coding may not be helping as much as you think(garymarcus.substack.com)
garymarcus.substack.com
AI coding may not be helping as much as you think
https://garymarcus.substack.com/p/breaking-news-ai-coding-may-not-be
2 comments
OP here—I'm posting this mostly to see what others think, but I have some serious problems with this study.
What happens when they break down the tasks? This study measures "development tasks."
Also, the study only mentions "245 tasks in mature projects" but doesn't specify what these tasks were anywhere. Were they debugging, new feature development, code review, architecture design, or simple maintenance? The tasks were in "mature projects" where developers had 4.9 years of experience on average.(Average residence at FAANGs is about 2-3 years before engineers move on...is that still true in 2025?)
That makes me wonder if these don't generalize to most real-world development work:
"experienced open-source developers"
"AI lab research engineer" jobs referenced
Those also make me wonder what kind of roles these are. They don't sound like general roles.
Another thing: Cursor Pro is a very specific tool. What about the time developers spent learning to use the AI tools during the study period (before they reached performance goals)?
What happens when they break down the tasks? This study measures "development tasks."
Also, the study only mentions "245 tasks in mature projects" but doesn't specify what these tasks were anywhere. Were they debugging, new feature development, code review, architecture design, or simple maintenance? The tasks were in "mature projects" where developers had 4.9 years of experience on average.(Average residence at FAANGs is about 2-3 years before engineers move on...is that still true in 2025?)
That makes me wonder if these don't generalize to most real-world development work:
"experienced open-source developers"
"AI lab research engineer" jobs referenced
Those also make me wonder what kind of roles these are. They don't sound like general roles.
Another thing: Cursor Pro is a very specific tool. What about the time developers spent learning to use the AI tools during the study period (before they reached performance goals)?
Are you saying that you think the tools are better than the study suggests or worse?
For all the “AI will be writing 100% of code by the end of this year claims”. Man I’m bought into the dream, I just wish it wasn’t so disappointing.