Training AI to Paint with Code using Reinforcement Learning

AI-Generated Art as Editable Code

Surya Narreddi and a research team have developed a method to train language models to create images by writing code rather than generating pixels. By treating the code as the primary artifact, the system allows for granular, direct editing of the image—a significant departure from traditional AI image generators where the only point of interaction is the prompt.

The Reinforcement Learning Loop

The system employs a four-step iterative loop run thousands of times during training to refine the model's ability to produce aesthetic results:

Generation: The model receives a prompt (e.g., "draw a peach hibiscus in watercolour") and writes a complete p5.brush JavaScript sketch.
Rendering: The sketch is executed in a sandboxed Puppeteer environment to produce a PNG image.
Judgment: A separate judge model compares the resulting PNG against two random reference paintings from a hand-rated pool and selects the superior watercolor.
Update: The judgment is converted into a reward signal, and Group Relative Policy Optimization (GRPO) is used to update the model.

Designing the Reward Function for Aesthetics

Because aesthetic quality is subjective and cannot be verified as "correct" or "incorrect" like a math problem, the project focuses on the design of the reward function as the primary challenge. The researchers found that the reward function must be carefully balanced: if it is too rigid, the model converges prematurely; if it is too loose, the model drifts.

The Reference Pool

To ground the reward signal, the team curated a pool of 581 reference paintings. These were derived from 1,664 generations and categorized into three tiers: 117 "love-tier," 266 "okay," and 198 supplements used to fill gaps in color representation.

Since human-made examples using the niche p5.brush library were scarce, the reference pool was populated using two pipelines:

AutoResearch: Utilized Opus 4.6, GPT-5.4, and Gemini 3.1 Pro to iterate against reference photographs under a VLM (Vision Language Model) judge.
Batch Run: A larger generation run using Gemini 3.1 Pro.

System Prompt Evolution and API Hallucinations

One of the key technical findings of the project is that providing extensive API documentation in the system prompt can actually increase hallucinations. Initial versions of the prompt included a 400-line p5.brush API reference, which led the model to confidently invent non-existent APIs.

To solve this, the team used GEPA (a prompt-optimization library) to evolve the prompt over 200 iterations against a taste-anchored 7-shot judge. The optimization converged on a highly constrained prompt featuring a strict allowlist of only eight brush methods and no API documentation or examples. This minimal, opinionated approach significantly reduced hallucinations and improved the actual visual output.

Synthesis of Community Insights

Community discussion around the project highlights the intersection of generative art and modern RL techniques. Some contributors noted the similarity to early research on learning brush strokes (e.g., GAN-based methods from 2018) and the potential for this approach to be applied to other formats like SVGs or voxels.

One technical counterpoint raised in the comments suggests that RL may be inefficient for generating style or novelty outside of the training distribution, noting that Supervised Fine-Tuning (SFT) on the base model often handles the bulk of the behavioral training for similar tasks:

"RL is very ineficient at style or at least at generating novelty out of distrib."

Conclusion

While generating images via code is slower than traditional diffusion models, it transforms the user from a spectator into a creative participant by providing an editable artifact. The project demonstrates that RL for creative tasks is essentially a design problem: creating a structure that allows human taste to generalize into model preferences.