The Dangerous Illusion of AI Coding: Jeremy Howard on Software Engineering vs. Coding

The Core Thesis: Coding is Not Software Engineering

AI-assisted coding creates a dangerous illusion of productivity by treating software development as a style-transfer problem rather than an engineering discipline. While Large Language Models (LLMs) can interpolate between existing patterns in their training data to generate syntactically correct code, they cannot perform the high-level design, architectural reasoning, and novel problem-solving that define true software engineering. This distinction matters because delegating cognitive tasks to AI without maintaining a deep mental model of the system leads to "understanding debt," where organizations possess working code that no human actually understands.

The Interpolation Illusion and "Vibe Coding"

Modern AI coding tools often function as a "slot machine," where developers pull a lever (the prompt) and receive a stochastic result. This process, termed "vibe coding," creates an illusion of control through prompt engineering and context management, but it lacks a grounding in technical intuition.

Compositional Creativity vs. True Innovation

Jeremy Howard distinguishes between different types of creativity:

• Combinatorial/Compositional Creativity: LLMs are highly effective at this. They can combine existing pieces of human knowledge in vast, new ways. For example, writing a C compiler in Rust is often an interpolation between existing Rust code and existing compiler logic in the training data, rather than a "clean room" invention.
• Extrapolative Creativity: LLMs struggle to move outside their training distribution. When a problem requires a truly novel solution that hasn't been designed before, LLMs often fail catastrophically, reverting to surface-level similarities that are fundamentally incorrect.

The Slot Machine Effect

Because LLMs provide occasional "wins" (working features), developers can become addicted to the loop of prompting and tweaking. Howard notes that this can be draining and stressful because the developer is no longer in control of the logic, but is instead gambling on the model's output.

The Risk of Organizational Enfeeblement

Delegating the act of coding to AI risks eroding the very knowledge an organization needs to survive and evolve. Howard argues that knowledge is not a static asset found on Wikipedia, but a living, embodied process of adaptive acquisition.

The Erosion of Control

When AI generates a majority of a codebase, a "pernicious erosion of control" occurs. Developers stop exercising their design and engineering muscles, leading to a state of "enfeeblement" where they lose the ability to grow in competence. This is particularly dangerous for mid-level developers who may never develop the deep intuition required to verify AI output.

Understanding Debt

Howard shares an example of using AI to fix a complex crash in IPykernel. While the AI eventually solved the problem, the result was a piece of code that no one understood. This creates a critical risk: if the code is not understood, its long-term stability, memory leaks, and edge-case vulnerabilities remain unknown. Betting a company's product on code that no human understands is a speculative and dangerous gamble.

Learning Through Friction: The Case for Interactive Programming

True technical intuition is built through "desirable difficulty"—the friction that occurs when reality pushes back against a mental model.

The Failure of Low-Friction Tools

Tools that remove all friction from the coding process (like autonomous AI agents) prevent the formation of long-term memories and deep understanding. Howard references the work of Ebbinghaus and the concept of spaced repetition, noting that memories are only formed when it is hard work to retrieve them.

The Power of the REPL and Notebooks

To combat enfeeblement, Howard advocates for a return to interactive, stateful environments (REPLs, Jupyter Notebooks). He argues that the ability to manipulate objects in real-time—poking at a problem until it pushes back—is where real insight happens.

"A human can do a lot more with a computer when the human can manipulate the objects inside that computer in real time, study them, move them around, and combine them together."

He emphasizes that the goal should not be to automate the human out of the loop, but to use AI within an interactive environment to enhance the human's ability to build a deeper mental model of the system.

AI Risk: Centralization vs. Extinction

Regarding existential risk, Howard argues that the primary danger is not an autonomous AI "taking off" and destroying the world, but the centralization of power.

• The Centralization Trap: Fear-mongering about AI extinction often leads to the call for centralizing power within a few large corporations or governments to "ensure safety."
• The Real Threat: Centralizing the most powerful technology in history into the hands of a few creates a massive incentive for power-hungry individuals to monopolize that control, which is a far greater risk to civilization than a rogue AI.

Howard believes the solution is to spread the technology across society, ensuring a balance of power and countervailing defenses rather than creating a single point of failure.