Training a coding model to paint watercolours with TRL and OpenEnv

Hugging Face has released an open reproduction of a project that trains a coding model to generate watercolor paintings by writing JavaScript code. By utilizing TRL and OpenEnv, the project demonstrates that Reinforcement Learning (RL) can be applied to "taste" and aesthetic preference, allowing a model to move beyond statistically average images toward a specific artistic style.

RL Over Aesthetic Taste

The core objective of this project is to determine if RL can be used to optimize for aesthetic preference rather than verifiable facts. The reward function is a weighted mix of four components designed to steer the model toward a specific watercolor style:

  • Pairwise Judge (60%): A vision model (Qwen3-VL-30B-A3B-Instruct) compares the candidate painting against four references randomly selected from a hand-curated pool. This encodes the specific taste of the curator.
  • HPSv3 (30%): An open 7B preference model that scores how much a general human population would prefer the image based on a text description.
  • Gate (5%): A binary check ensuring the sketch compiles, uses the allowed library, and produces actual pigment without "cheating" (e.g., writing text on the canvas).
  • Length (5%): A soft push toward longer code snippets.

Three training runs were conducted to test the influence of these judges:

Run Pairwise Judge Weight HPSv3 Weight Result
judge-led 0.60 0.30 Most diverse and artistically interesting results.
hps-led 0.30 0.60 Convincing watercolors with a consistent "wet-on-wet" look.
hps-only 0.00 0.90 Reliable but converged quickly to similar colors and styles.

The Technical Environment and Constraints

The training pipeline relies on a specialized RL environment that wraps the model's output and converts it into a visual render.

p5.brush Library

Instead of drawing basic shapes, the model uses the p5.brush library, which simulates physical watercolor properties such as pigment bleed, paper texture, and flow fields. To maintain the watercolor aesthetic, the model is restricted to a strict allowlist of only 10 methods (e.g., fillBleed, fillTexture, beginShape).

The Reference Pool

Taste is defined by a reference pool of 178 paintings, divided into love and okay tiers. These images were generated by four open-weight models (GLM-5.2, Kimi-K3, Qwen3-Coder-Next, and Qwen3.5-122B-A10B) and refined through a vision critic before being hand-rated. During training, the pairwise judge draws references from both tiers to ensure the model receives a signal even in early, weak stages of policy development.

Training Implementation and Optimization

The project used the GRPOTrainer from TRL and the Qwen/Qwen3.5-35B-A3B model. Several critical configuration changes were required to unlock learning:

  • Learning Rate: Increased from 2e-5 to 5e-5.
  • Scheduler: Changed from linear to constant_with_warmup to prevent the learning rate from decaying too early.
  • Reward Scaling: Set scale_rewards to none to prevent a single gate rejection from shrinking the advantages of the entire group.
  • LoRA Target Modules: Changed to all-linear to ensure the adapter reached every linear layer in the MoE architecture.

Key Findings and Learning Outcomes

Distribution Shift

In all runs, the model first learned to eliminate "bad" paintings (near-blank canvases or shapeless washes). In the hps-only run, the improvement was primarily in the median of the distribution. However, in the judge-led and hps-led runs, the top-end quality also increased, with paint coverage doubling as the model was rewarded for matching the curated pool's style.

Constraints vs. Rewards

The model learned to ignore explicit instructions in the system prompt that were not tied to the reward. For example, a request to produce 15-30 filled shapes was ignored in favor of producing 7-9 shapes, as the number of shapes did not correlate with the reward.

Infrastructure and Costs

The pipeline is hosted end-to-end on Hugging Face, utilizing HF Jobs for training, Spaces for the RL environment and HPSv3 scorer, and Inference Providers for the pairwise judge.

  • Compute: A single run of 110 steps took approximately 34 hours on one H200 GPU.
  • Bottleneck: Rendering is the primary time sink, accounting for 70-80% of the step time because the headless Chromium renders the WEBGL canvas in software on a CPU-based Space.

Future Directions

The author identifies several potential improvements for future iterations:

  • Multi-step Feedback: Implementing a loop where the model can see its own painting and refine it, similar to how the reference pool was created.
  • Model Scaling: Testing smaller models (e.g., 4B), as initial experiments suggest they can produce valid sketches.
  • SFT Pre-training: Performing Supervised Fine-Tuning on the pool sources before starting RL.

Sources