Pelican-bicycle alternatives 2026 benchmark reveals SVG generation progress across major LLMs

Quick takeaway

The updated 2026 Pelican‑bicycle SVG benchmark demonstrates that flagship models from OpenAI, Anthropic, Google, DeepSeek, Alibaba, and others now generate coherent, detailed vector graphics for absurd prompts, with only modest differences in latency and cost, while community discussion highlights emerging cost‑performance trade‑offs and style homogenisation.

What the benchmark measures

Prompt: A short, whimsical description (e.g., “an octopus operating a pipe organ”).
Output: An SVG image generated by each model.
Metrics reported: Generation time and API cost per request.
Model sets: Two runs – a 2026 run (6 models) and a 2025 run (9–10 models) covering GPT‑6 Astra, Claude Fable 5.1, Gemini 3.8 Flash, DeepSeek V4 Pro, Qwen 3.8 Max, and Fugu Ultra v2, plus a broader 2025 roster (Claude Sonnet 4.5, Claude Opus 4.5, GPT‑5.1, GPT‑5.2 Pro, Gemini 2.5 Pro, Gemini 3.0 Pro Preview, Grok Code Fast 1, DeepSeek V3.2‑Exp, GLM‑4.6, Qwen3‑VL‑235B‑A22B‑Thinking).

Performance snapshot by model family

Model family	Typical latency (2026 run)	Typical cost (2026 run)	Notable visual traits
OpenAI GPT‑6 Astra	1–2 min	$0.20‑$0.37	Clean line work, consistent anatomy
Anthropic Claude Fable 5.1	1–3 min	$0.46‑$0.79	Slightly richer color palette
Google Gemini 3.8 Flash	2–5 min	$0.07‑$0.12	Often more vibrant but occasionally blurry
DeepSeek V4 Pro	1–5 min	$0.04‑$0.10	Minimalist style, low cost
Alibaba Qwen 3.8 Max	9–15 min	$0.15‑$0.27	Higher latency, competitive cost
Sakana AI Fugu Ultra v2	2–14 min	$0.35‑$2.05	Highest cost, variable detail

Observation: Gemini 3.8 Flash offers the best performance‑to‑cost ratio for many prompts, as noted by a community comment.

Cost‑performance trends

Latency has generally decreased for newer models (e.g., GPT‑6 Astra ~2 min vs. GPT‑5.1 ~2 min in 2025) while maintaining similar quality.
API cost is now clustered around $0.05‑$0.30 for most models, with outliers like Fugu Ultra v2 (up to $2.05) and DeepSeek V4 Pro (as low as $0.04).
Community member @BrokenCogs highlighted Gemini 3.8 Flash as an outlier with a strong cost‑performance profile.

Visual quality insights from the discussion

Style convergence: Multiple commenters (e.g., @outlore, @dustfinger) observed that different models produce remarkably similar compositions—e.g., the moose consistently stands on the left side of the carousel across providers.
Anatomical challenges: @samayashar pointed out lingering issues where models mis‑place limbs (the octopus’s tentacle emerging from the organ rather than the body).
Color and vibrancy: @sajithdilshan praised Gemini 3.8 for its more vivid palette compared with other providers.
Cost‑effective quality: @andy_ppp expressed surprise at Qwen 3.8’s impressive output relative to its price, echoing the earlier cost‑performance note.
Benchmark saturation: @svcrunch referenced the Little Dorrit Benchmark, noting that top scores are still far from saturation (≈0.78 F1), implying room for further differentiation.

Limitations of the benchmark

Goodhart’s Law: As @vova_hn2 warned, the benchmark may be over‑fit; models could have memorised similar SVG tasks from training data, reducing its ability to probe emergent reasoning.
Lack of negative‑space handling: @theshrike79 observed that SVG outputs avoid the “fill‑everything” artifact common in raster generators, but this also masks a potential weakness in handling empty regions.
Missing prompts: Several 2026 prompts (e.g., sloth steering an excavator, dragonfly balancing a chandelier) have not yet been generated, limiting a full 2026 comparison.

Community‑driven ideas for future extensions

Animated SVGs: @qiine suggested adding an animation dimension to increase difficulty.
Origami folding tasks: @eddytrex_ proposed an SVG‑based origami benchmark.
Parameter‑sweep for cheap models: @miohtama hopes for iterative inspection/fixing loops to improve low‑cost model outputs.

What’s next for SVG‑generation benchmarking?

Broader prompt diversity – introduce tasks that require explicit negative‑space reasoning or multi‑step composition.
Dynamic evaluation – automate F1‑style scoring (as in the Little Dorrit Benchmark) to quantify alignment quality.
Cost‑normalized ranking – publish a leaderboard that normalises quality by latency and API price, making trade‑offs transparent.
Open‑source reference implementations – provide a baseline SVG renderer to verify that generated code is syntactically valid and renders consistently across browsers.

Final assessment

The 2026 Pelican‑bicycle SVG benchmark confirms that mainstream LLMs have reached a maturity level where whimsical vector graphics can be generated reliably, with latency and cost now comparable across vendors. However, the community’s observations of style homogenisation, lingering anatomical glitches, and potential benchmark over‑fitting suggest that future work should focus on richer, cost‑aware evaluation metrics and more challenging compositional tasks.