Claude Fable 5 vs GPT-5.6 Sol $100 AI Music Video Build‑off

TL;DR

Claude Fable 5 and GPT‑5.6 Sol were each given a $25 or $100 budget, a song (Uptown Funk), and a set of six tools, and they autonomously generated complete music videos. All four runs finished, but the videos were low‑quality, showed poor story consistency, and highlighted gaps in frontier‑level LLMs for long‑horizon creative tasks.

Experiment Overview

Goal: Test how frontier LLMs use tools when tasked with an open‑ended, long‑horizon project – creating a music video from scratch.

Setup:

  • Input: Bruno Mars & Mark Ronson’s Uptown Funk, a short textual description, and a time‑stamped lyric transcript.
  • Harness: An autonomous loop that allowed the model to call six tools:
    1. plan – internal reasoning (free).
    2. web_search – fetch information about video generation models.
    3. get_budget – query remaining dollar budget.
    4. generate_image / generate_video – spend budget on FAL or Replicate models.
    5. run_command – local shell with ffmpeg for analysis, cutting, concatenation, and final muxing.
  • Budget caps applied only to the generation tools; editing continued after the budget hit zero.
  • All tool calls, costs, and errors were logged. The harness is open‑source at github.com/hershalb/music-video-arena.

Four runs were performed:

Model Budget Video Link
Claude Fable 5 $25 Video 9
GPT‑5.6 Sol $25 Video 10
Claude Fable 5 $100 Video 11
GPT‑5.6 Sol $100 Video 12

All videos are full‑length MP4s with the original song muxed in.

Performance Metrics

Model Budget Wall‑clock time Steps Failed calls Generation spend Resolution
Claude Fable 5 $25 39 m 10 s 25 0 $24.30 1280×720
GPT‑5.6 Sol $25 42 m 52 s 38 61 $23.18 1280×720
GPT‑5.6 Sol $100 49 m 39 s 34 0 $36.57 1280×720
Claude Fable 5 $100 38 m 56 s 28 0 $48.60 1920×1080

Generation spend is the metered FAL cost, which the budget caps. Token‑model execution cost is additional (see “Total cost per run”).

Tool‑Use Patterns

Run Image model Video model(s) Approach
Fable 5 · $25 none Wan 2.5 t2v ($0.05/s) Pure text‑to‑video
Sol · $25 FLUX schnell ($0.003/img) Wan 2.2‑5b i2v ($0.10/s) Keyframe → image‑to‑video
Sol · $100 none Wan 2.5, Veo 3.1 Lite, Hailuo 2.3 Standard Mixed text‑to‑video models
Fable 5 · $100 none Seedance 1.0 Pro t2v (~$0.12/s) Pure text‑to‑video

Distinct clips generated per run ranged from 46 to 80. Only GPT‑5.6 Sol at $25 used an image‑to‑video pipeline; the other three runs relied solely on text‑to‑video models.

Cost Breakdown

Run Generation spend LLM token cost* Total cost
Fable 5 · $25 $24.30 $16.99 $41.29
Sol · $25 $23.18 $4.27 $27.45
Sol · $100 $36.57 $3.25 $39.82
Fable 5 · $100 $48.60 $25.05 $73.65

*Claude Fable 5 token pricing: $10 / $50 per 1 M input/output tokens. GPT‑5.6 Sol token pricing: $5 / $30 per 1 M tokens.

Key observations:

  • At the $25 level both models almost exhausted their generation budget.
  • At $100 the models spent $36.57 (Sol) and $48.60 (Fable), showing that additional budget translates into more footage but not proportionally.
  • Claude Fable 5’s token cost accounted for 30‑40 % of its total run cost, whereas GPT‑5.6 Sol’s token cost stayed under $5 regardless of budget.

Creative Outcomes

Common Shortcomings

  • Story & character consistency: Characters drift between shots; no coherent narrative across the video.
  • Literal lyric interpretation: The models render lyrics verbatim (e.g., a literal dragon “wanna retire”), leading to odd visual jokes.
  • Tempo mismatch: Beat detection aligns cuts to the beat, but the motion within clips rarely matches the song’s tempo, producing a “off‑beat” feel.
  • Lack of iteration: Once clips were generated, models concatenated them without revisiting or refining earlier footage.
  • No Replicate usage: Despite having Replicate API keys, all runs used FAL models exclusively.

Model‑Specific Findings

  • GPT‑5.6 Sol $25 was the most inventive editor, overlaying text and animating stills—techniques absent from the other runs.
  • GPT‑5.6 Sol $100 mixed three video models, showing willingness to experiment with multiple generators.
  • Claude Fable 5 consistently chose a single video model per run, resulting in more uniform visual style but higher token cost.
  • Claude Fable 5 $100 produced the highest‑resolution output (1920×1080) and finished fastest (38 m 56 s), yet its total cost was the highest at $73.65.

Community Reaction (Hacker News Highlights)

"These videos are awful. It’s like Suno music. As soon as you pay attention you notice all the cracks." – saaaaaam

"GPT‑5.6 Sol at $25 was the most inventive editor… the rest mostly just stitched generated clips together." – author’s take

"The result depends mostly on the LLM’s ability to come up with a good story, choose and prompt the video model, evaluate and edit. The LLMs mostly used older video models capped at 10‑second clips." – yanexr

"Token cost for the $100 video was $3.25 for Sol 5.6 and $25.05 for Fable 5 – an interesting benchmark of efficiency." – spongebobstoes

"These videos have the ‘karaoke’ vibe…" – tarosnow

The comments range from disappointment in artistic quality to curiosity about cost efficiency and speculation on future workflows.

Lessons Learned

  1. Tool selection matters: GPT‑5.6 Sol explored multiple video generators, while Claude Fable 5 stuck to one, affecting visual diversity.
  2. Budget does not guarantee usage: Even with $100, models left substantial budget unused, indicating limited internal budgeting heuristics.
  3. Self‑review is missing: Neither model re‑evaluated generated clips before final assembly, a gap that human editors fill.
  4. Cost efficiency varies widely: GPT‑5.6 Sol achieved comparable video length for a fraction of Claude Fable 5’s token cost.
  5. Current video models are a bottleneck: Most runs relied on older 10‑second clip models; newer longer‑clip models (e.g., Seedance 2) could improve continuity.

How to Replicate

  1. Clone the open‑source arena: git clone https://github.com/hershalb/music-video-arena.
  2. Provide your own song, description, and lyric timestamps.
  3. Set a dollar budget and choose which LLMs to pit against each other.
  4. Run the harness; it will log every plan, tool call, and ffmpeg command.
  5. Review the generated transcripts (e.g., fable-25.txt, sol-100.txt) for deeper insight.

Final Verdict

While the videos are far from broadcast‑ready, the experiment demonstrates that frontier LLMs can autonomously orchestrate a multi‑tool pipeline, manage a budget, and produce a watchable output. The creative shortcomings—literal lyric handling, poor pacing, and lack of iterative refinement—highlight the remaining gap between raw generation capability and human‑level artistic direction.

All models mentioned are available on TryAI with pay‑as‑you‑go pricing. You can start a free trial at https://www.tryai.dev/chat.

Sources

Related