How AI assistance impacts the formation of coding skills

Anthropic research indicates that using AI assistance during the learning process leads to a statistically significant decrease in skill mastery, specifically reducing quiz scores by 17% compared to those who code by hand. While AI can accelerate the completion of familiar tasks, it may undermine the acquisition of new skills and the ability to debug code, creating a risk where developers lack the expertise necessary to oversee AI-generated output.

Impact on Skill Acquisition and Mastery

Using AI assistance during the learning of a new technical skill results in lower comprehension and retention. In a randomized controlled trial involving 52 mostly junior software engineers learning the Python library "Trio," the AI-assisted group averaged a 50% score on a post-task quiz, while the hand-coding group averaged 67%. This difference is equivalent to nearly two letter grades (Cohen's d=0.738, p=0.01).

Key findings regarding performance include:

Debugging Gap: The largest disparity in scores occurred in debugging questions, suggesting that AI reliance specifically hinders the ability to identify and diagnose errors.
Productivity Trade-off: The AI group finished tasks approximately two minutes faster on average, but this productivity gain was not statistically significant.
Cognitive Offloading: Participants using AI often offloaded their thinking to the tool, reducing the effort put into the task and decreasing engagement with the work.

AI Interaction Patterns and Learning Outcomes

Qualitative analysis of screen recordings reveals that the method of interacting with AI determines whether a user retains information. The study identified two distinct categories of interaction patterns:

Low-Scoring Interaction Patterns (Average < 40%)

AI Delegation: Users wholly relied on AI to write code. This group was the fastest to complete tasks and encountered the fewest errors but showed the lowest mastery.
Progressive AI Reliance: Users started with independent effort but eventually delegated all code writing to the AI.
Iterative AI Debugging: Users relied on AI to solve problems and verify code rather than using the tool to clarify their own understanding.

High-Scoring Interaction Patterns (Average ≥ 65%)

Generation-then-Comprehension: Users generated code and then asked follow-up questions to build understanding.
Hybrid Code-Explanation: Users requested code generation and detailed explanations simultaneously.
Conceptual Inquiry: Users asked only conceptual questions and relied on their own understanding to complete the task. This was the second fastest overall pattern and associated with high mastery.

Implications for Software Engineering and AI Policy

The study suggests that aggressive AI integration in the workplace may create a long-term risk to societal resilience and system safety. If junior engineers' skill development is stunted, they may lack the capacity to provide meaningful oversight or catch errors in high-stakes environments where AI-generated code is deployed.

To mitigate these risks, the research suggests:

Intentional Skill Development: Encouraging "cognitive effort" and the process of getting stuck as a necessary part of fostering mastery.
Tool Design: Utilizing learning modes (such as Claude Code Learning and Explanatory mode or ChatGPT Study Mode) designed to foster understanding rather than just output.
Management Strategy: Designing AI deployment policies that ensure engineers continue to learn while they work, rather than prioritizing immediate speed over long-term expertise.

Study Context and Limitations

This research focuses specifically on the acquisition of new skills, contrasting with previous Anthropic observational data showing AI can speed up existing tasks by 80%. The study notes that AI may simultaneously accelerate productivity for well-developed skills while hindering the acquisition of new ones.

Limitations of the study include a relatively small sample size and the measurement of comprehension immediately after the task, leaving the question of whether immediate performance predicts long-term skill development open for further research.