Are We Letting AI Code for Us — and Killing Our Skills?
The Trade-off Between Mastery and Speed. With so many overblown headlines, it's challenging to get a clear grasp of the productivity improvements that AI coding tools like ChatGPT, GitHub Copilot, and Cursor truly bring. At our studio, we've found these AI coding tools offer noticeable productivity gains. While some claim a 10x increase, our experience aligns more with a modest yet significant improvement. In fact, a recent Google study estimates an average 21% improvement in developer productivity for AI users. We're not in the strongly anti-AI camp. On the contrary, we believe it has made our engineers — most of whom are senior — more productive. Yet, at the same time, we've grown concerned about the long-term effects of over-dependence on AI tools. In the long run, will it make us worse at writing software? Does it deprive us of practice, causing us to learn more slowly?
Skill Atrophy: Is There a Risk?
We don't know the answer for certain, but a recent study on essay writing gives us pause. A draft paper published earlier this month by MIT suggests that when writing an essay, those using ChatGPT "consistently underperformed" compared to those using a search engine or nothing at all. They "significantly underperformed" when asked to quote from the essay they produced.
Overall, those who didn't use ChatGPT "engaged more extensive brain network interactions"; perhaps not surprisingly, the brains of the non-AI group worked harder. While writing code is different enough from writing essays that we should be cautious about drawing too many conclusions from this study, it is a cause for concern.
Of course, how we use AI tools matters a lot. Presumably, if we ensure we understand every line of code generated for us, we're doing better than anyone who blindly trusts what the tools have produced. The AI-assisted essay writers in the study seemed to delegate most of the thinking to ChatGPT, and we don't have to use AI tools in this way.
Nevertheless, when taking suggestions from AI, we become more like code reviewers and less like active creators. Does this matter? We haven't found any studies specifically answering this question, but we suspect that while code review certainly teaches the reviewer something, this learning is unlikely to be as deep as if we wrote the code ourselves.
Speed Versus Deep Understanding
On the flip side, AI tools can be a significant time-saver. In the past, one of our engineers would probably have gone to StackOverflow to grab a good implementation of a debounce function, as they'd likely find something better than what they could come up with themselves. Now, Cursor autocompletes it. (And, for the record, we think that's fine: you don't have to have a perfect debounce function committed to memory to be a good software engineer).
Occasionally, we trust Cursor with larger tasks, such as writing unit tests for a particular chunk of code. The output almost always requires tweaking, but — if we can provide an example of something similar — the result is usually a decent starting point, saving us time that we would have spent writing boilerplate or tweaking an existing test file to fit a new purpose.
For professional programmers, productivity matters, and particularly for our senior engineers, we're measured on both the quality and quantity of our output. But learning matters too. Regardless of whether LLMs' coding ability plateaus or continues to grow, we believe there will always be a market for human beings who deeply understand code. If we want to be (or continue to be) those people, we should think carefully about how much of our thinking we delegate to AI.
As so often, balance is probably the best approach. Throwing out AI tools entirely may do more harm than good, especially in a job market where "AI skills" are often high on the agenda. But it's important to know when and how to use it: don't delegate away too much of the important thinking to AI.