Are AI Labs Pelicanmaxxing? Evidence from a 1,008‑SVG Experiment

Introduction

The takeaway is that, after generating 1,008 SVGs across seven leading models and scoring them with an LLM judge, there is no clear evidence that any lab is optimizing specifically for the pelican‑on‑a‑bicycle benchmark. The pelican‑bicycle cell does not outperform what the model’s general animal and vehicle abilities predict, and any observed effects are within random variation.

Methodology

The experiment tested seven frontier models accessed via OpenRouter: GPT‑5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7‑Max, GLM‑5.2, and DeepSeek V4 Pro. For each of 8 animals × 6 vehicles = 48 prompts, three samples were generated at temperature 1.0, yielding 1,008 SVGs. SVGs were rendered to PNG; any failed render triggered a retry until a valid image appeared, with only 11 retries total. Each image was scored by GPT‑5.6 Luna on animal, vehicle, and coherence (1‑5). The average of the three scores formed the judge score. Gemini 3.1, Gemini 3.1 Flash‑Lite extracted recognized animal, vehicle, facing direction, and scene elements to assess memorization.

Evidence 1: Visual inspection shows no obvious pelican‑bicycle advantage

Looking at the raw images, no pelican‑bicycle drawing appeared noticeably better than the rest of that model’s grid. The author noted that, while GLM‑5.2’s first pelican‑bicycle sample looked slightly better, the same batch also produced a strong heron‑on‑skateboard, making the impression ambiguous. Overall, labs that draw good pelicans on bicycles also draw other animal‑vehicle combinations well.

Evidence 2: Labs are not better at drawing pelicans

Pooled animal ratings placed the pelican sixth out of eight animals, behind cat, whale, raccoon, heron, and antelope. If labs were training on the pelican benchmark, pelicans would rank higher. The ranking alone does not rule out difficulty effects, but it shows pelicans are not advantaged relative to other animals.

Evidence 3: Labs are not better at drawing bicycles

Bicycle ratings ranked second‑to‑last, just above planes. A lab optimizing for bicycles would push them toward the top. The low score reflects the inherent difficulty of drawing two aligned wheels, a frame, handlebars, seat, and pedals; many bicycle images miss or disconnect parts. The plane vehicle suffered from ambiguous prompts (“plane” interpreted as a flat surface), but bicycles still did not show a lab‑specific boost.

Evidence 4: Adjusted regression finds no significant pelican‑bicycle boost

A fixed‑effects regression (score ~ lab + animal × vehicle) with per‑lab interaction terms for pelican, bicycle, and the pelican‑bicycle cell was run on all 1,008 images. Results:

Pelican effects per lab ranged from –0.11 to +0.14 judge points; none were significant (smallest p = 0.25).
Bicycle effects per lab ranged from –0.18 (Grok 4.5, p = 0.11) to +0.27 (Gemini 3.5 Flash, p = 0.022). Only Gemini cleared p < 0.05, but with 21 tests the expected false‑positive count is about one, and the result does not survive Bonferroni correction (threshold ≈ 0.002).
Pelican‑bicycle interaction effects were all non‑significant; the largest positive was GLM‑5.2 at +0.35 (p = 0.12). All confidence intervals included zero, indicating no detectable lab‑specific advantage beyond what animal and vehicle main effects predict.

Evidence 5: Scene composition and memorization do not reveal pelican‑bicycle overfitting

All 21 pelican‑bicycle images faced right; no other animal‑vehicle combination showed unanimous direction. However, facing right is common overall (60 % of images) and especially prevalent for bicycles (81 % right) and pelicans (78 % right). Other combinations such as antelope‑on‑scooter and heron‑on‑bicycle also reach 19‑21 right‑facing samples, so the 21‑out‑of‑21 result is not an outlier. Feature extraction showed recurring elements for some pairs (e.g., every flamingo‑on‑boat includes a sun; otters‑on‑plane wear scarves 38 % of the time; cats‑on‑bicycle get a basket 38 % of the time). Pelican‑on‑bicycle had no uniquely frequent element set; its scene elements varied like other pairs.

Limitations

The study relied on a single LLM judge (GPT‑5.6 Luna) for scoring; inter‑judge reliability was not measured.
A lab that improves overall SVG generation (SVGmaxxing) would lift all cells equally, making it indistinguishable from general competence in this design.
Budget limited the experiment to three samples per cell, one judge, and seven models, preventing extensive prompt iteration or additional models.

Discussion of community comments

Commenters highlighted several points that align with the analysis:

Simon Willison noted the robustness of generating 1,008 SVGs across an 8×6 grid and agreed that nothing jumped out as a pelican‑bicycle advantage.
mauvehaus explained the right‑facing bias by noting that bicycle drivetrains are on the right side, so training data likely shows bicycles from that perspective, which accounts for the directional uniformity without implying memorization.
stusmall welcomed the quantitative approach, countering the common dismissive claim that labs "train on it by now."
Wowfunhappy suggested that improving general SVG drawing (SVGmaxxing) is a more plausible explanation for any observed gains.
bnfcl reported a similar experiment checking whether models default to pelican‑on‑bicycle when no specifics are given, finding no such tendency.
elliotto reinforced the drivetrain explanation, citing common bike‑photography practice.
Some commenters questioned the use of an LLM judge, but the author argued that any systematic judge bias would shift all labs equally and thus not affect within‑lab contrasts.

Conclusion

The evidence does not support the hypothesis that AI labs are pelicanmaxxing the pelican‑on‑a‑bicycle benchmark. Pelicans and bicycles are not drawn better than other animals and vehicles, and the specific combination does not exceed what the model’s baseline abilities predict. The most plausible story is that any improvements lie in general SVG capability rather than targeted benchmark gaming.