Are AI Labs Pelicanmaxxing? Evidence from a 1,008‑SVG Experiment

Takeaway

The analysis of 1,008 SVGs from seven frontier models shows no lab that draws pelicans on bicycles better than its baseline animal and vehicle performance predicts. After adjusting for inherent difficulty, the pelican‑bicycle cell effects are small, non‑significant, and explainable by chance.

Experimental Design

The study tested seven models via OpenRouter: GPT‑5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7‑Max, GLM‑5.2, and DeepSeek V4 Pro. For each of 8 animals × 6 vehicles = 48 prompts, three samples were generated at temperature 1.0, yielding 1,008 SVGs. The prompts followed Simon Willison’s phrasing, varying only the animal and vehicle. Each SVG was rendered to PNG; failed renders were retried until a valid image appeared (only 11 retries total). Scores came from a three‑stage pipeline: (1) rendering, (2) judgment by GPT‑5.6 Luna giving 1‑5 ratings for animal, vehicle, and coherence, averaged into a judge score, and (3) feature extraction by Gemini 3.1 Flash‑Lite recording recognized animal, vehicle, facing direction, and scene elements.

Results: Animal and Vehicle Ratings

Pelicans ranked sixth of eight animals in mean rating, behind cat, whale, raccoon, heron, and antelope. Bicycles ranked second‑last of six vehicles, near‑tied with planes. These positions indicate that, on average, the models do not draw pelicans or bicycles better than other categories. A lab that trained specifically on the benchmark would be expected to push these items to the top of the rankings, which did not occur.

Regression Analysis

A fixed‑effects regression modeled score as score ~ lab + animal × vehicle, plus per‑lab interaction terms for pelican, bicycle, and the pelican‑bicycle cell, with robust standard errors. The animal × vehicle terms captured baseline difficulty of each combination. The interaction terms measured each lab’s benchmark‑specific boost relative to the average lab.

• Every per‑lab pelican effect lay between –0.11 and +0.14 judge points; none reached significance (smallest p = 0.25).
• Per‑lab bicycle effects ranged from –0.18 (Grok 4.5, p = 0.11) to +0.27 (Gemini 3.5 Flash, p = 0.022). Only Gemini cleared p < 0.05, but with 21 tests the Bonferroni threshold is ≈0.002, so this does not survive multiple‑comparisons correction.
• No pelican‑bicycle cell effect exceeded p < 0.05; the largest positive was GLM‑5.2 at +0.35 (p = 0.12). All confidence intervals for pelican, bicycle, and cell effects included zero, indicating no reliable boost.

Direction and Scene Elements

All 21 pelican‑bicycle images, across all labs, faced right; no other animal‑vehicle combination showed unanimous direction. However, facing right is common overall (60 % of images) and especially strong for bicycles (81 % right) and for birds such as pelicans (78 % right). Commenters noted that bicycle drivetrains are conventionally photographed from the right side, which likely influences the models’ outputs. Scene‑element extraction found no recurring unique combination for pelican‑bicycle; elements such as sun, scarf, or basket appeared at rates similar to other pairs. One comment observed that the uniform right‑facing does not imply memorization because the same tendency appears across many combinations.

Limitations

• The study used a single LLM judge (GPT‑5.6 Luna) for scoring; inter‑judge reliability was not assessed.
• A lab that improved SVG generation broadly would lift all cells equally, making it indistinguishable from general competence (so‑called SVGmaxxing).
• Budget limited the experiment to three samples per cell, a single judge, and seven models, preventing extensive prompt iteration (e.g., using “airplane” instead of “plane”).

Discussion (Comment Insights)

Several commenters highlighted relevant points:

• Simon Willison expressed hope that a lab would be caught outperforming on the pelican‑bicycle benchmark, calling the experiment "significantly more robust" than his informal checks.
• Mauvehaus argued that the right‑facing bicycle bias stems from the drivetrain being on the right side, suggesting the models reflect common photographic conventions rather than benchmark‑specific training.
• Wowfunhappy noted that "SVGmaxxing" — improving general SVG ability — is a plausible explanation for strong performance across many prompts.
• Throwaway6s1df questioned the statistical soundness of aggregating directional biases, but the analysis already accounted for multiple comparisons.
• Others praised the null result as refreshing and emphasized the difficulty of detecting subtle benchmark‑specific tuning.

Conclusion

The evidence does not support the hypothesis that AI labs are pelicanmaxxing the pelican‑on‑bicycle benchmark in a plainly observable way. Pelicans and bicycles are not drawn better than other animals and vehicles, and the specific pelican‑bicycle combination does not show a significant boost after adjusting for difficulty. The uniform right‑facing orientation aligns with common visual conventions rather than indicating memorization. While broader SVG capability improvements (SVGmaxxing) may exist, the experiment cannot detect them due to its design. Thus, at present, there is little indication that labs are generating large volumes of pelican‑on‑bicycle SVGs solely to game a benchmarking to trick informal evaluators.