Claude Fable 5 vs GPT-5.6 Sol AI Music Video Arena: $25 and $100 Budget Comparison
Claude Fable 5 vs GPT-5.6 Sol AI Music Video Arena: $25 and $100 Budget Comparison
Quick Take
Claude Fable 5 and GPT-5.6 Sol were each given a $25 or $100 budget to autonomously create a music video for Bruno Mars & Mark Ronson’s Uptown Funk. All four runs completed without hitting step or time limits, but the videos were low‑quality, with notable differences in model‑chosen generation tools, spend efficiency, and editorial creativity.
Experiment Overview
The test harness gave each model a single job: receive the song, a hard dollar budget, and a toolbox, then produce a finished video without further human intervention. The toolbox consisted of six tools:
- plan – cost‑free reasoning step.
- web_search – optional research of video generation APIs.
- get_budget – query remaining budget.
- generate_image / generate_video – paid calls to any FAL or Replicate model.
- run_command – local shell with
ffmpeg/ffprobefor audio analysis, clipping, concatenation, and final muxing.
When the budget reached zero, further paid generation was blocked, but the model could continue editing.
All code is open‑source at github.com/hershalb/music-video-arena.
Results Summary
| Model | Budget | Wall‑clock time | Distinct clips | Generation spend (FAL) | LLM token cost | Total cost |
|---|---|---|---|---|---|---|
| Claude Fable 5 | $25 | 39m 10s | 54 | $24.30 | $16.99 | $41.29 |
| GPT‑5.6 Sol | $25 | 42m 52s | 46 | $23.18 | $4.27 | $27.45 |
| GPT‑5.6 Sol | $100 | 49m 39s | 70 | $36.57 | $3.25 | $39.82 |
| Claude Fable 5 | $100 | 38m 56s | 80 | $48.60 | $25.05 | $73.65 |
Generation spend reflects the metered FAL cost; the LLM token cost adds the price of running Claude ($10 / $50 per M tokens) or Sol ($5 / $30 per M tokens). The $100 runs did not exhaust their caps, indicating that the models were conservative with spending.
Tool‑Selection Strategies
| Run | Image model | Video model(s) | Approach |
|---|---|---|---|
| Claude Fable 5 – $25 | none | Wan 2.5 t2v ($0.05/s) | Pure text‑to‑video |
| GPT‑5.6 Sol – $25 | FLUX schnell ($0.003/img) | Wan 2.2‑5b i2v ($0.10/s) | Keyframe image generation → image‑to‑video |
| GPT‑5.6 Sol – $100 | none | Wan 2.5, Veo 3.1 Lite, Hailuo 2.3 Standard | Mixed text‑to‑video models |
| Claude Fable 5 – $100 | none | Seedance 1.0 Pro t2v (~$0.12/s @1080p) | Pure text‑to‑video |
Claude stuck to a single text‑to‑video model in both runs, while Sol experimented with an image‑to‑video pipeline at $25 and mixed three video models at $100. Neither model used Replicate APIs despite having keys.
Cost Efficiency
- At the $25 level both models nearly exhausted their generation budget.
- At $100, Sol spent $36.57 (≈ 36 % of the cap) while Claude spent $48.60 (≈ 49 %).
- Token costs were a larger share for Claude (≈ 30‑40 % of total) than for Sol (≈ 10 %).
- Overall, Sol produced the cheapest full run ($27.45) but with fewer clips and lower resolution (720p vs. Claude’s 1080p at $100).
Creative Outcomes
Consistency & Narrative
"Character and story consistency was a struggle for all four. Recurring characters drift between shots, and none of the videos hold a coherent storyline from start to finish." – Author’s take Both models treated lyrics literally, leading to odd literal visualizations (e.g., a dragon “wanna retire”). Narrative arcs were absent.
Tempo & Editing
"Tempo matching is weak. The cuts land on the beat, but the motion inside the clips rarely matches the song’s tempo, making the video feel off." All runs used ffmpeg beat detection for cut points, yet clip motion and dancing were often out of sync.
Inventiveness
- GPT‑5.6 Sol at $25 was the most inventive editor, overlaying text and animating stills with video effects—techniques absent from the other runs.
- GPT‑5.6 Sol at $100 tried multiple video models, whereas Claude kept a single model throughout.
Iteration & Self‑Review
"Nobody really iterated on the edit. Once clips existed, the models concatenated and muxed, but rarely went back to re‑cut or add effects." Both models lacked a feedback loop to evaluate generated clips before final assembly.
Error Handling & Reliability
Failed generation calls (mostly transient network errors) were logged but not charged. Models retried these calls, adding to wall‑clock time but not to the $ budget.
Community Reaction (Hacker News Highlights)
- Skepticism – Several commenters called the videos “the worst thing I have ever seen” and warned that such low‑quality output fuels anti‑AI sentiment.
- Appreciation for the experiment – Others noted the value of the test as a glimpse into future workflows, especially for bulk‑style content like ad inserts.
- Creative suggestions – A comment linked to a YouTube example that embraces the “uncanny‑valley” aesthetic as a feature rather than a bug.
- Human‑in‑the‑loop advocacy – One user shared a link to a higher‑quality AI music video that used human editing, underscoring the current need for manual refinement.
Lessons Learned
- Frontier models still lack high‑level storytelling – Literal lyric interpretation dominates.
- Tool‑use diversity matters – Sol’s mixed‑model approach yielded more varied footage, though not necessarily better quality.
- Budget caps are not fully utilized – Both models left significant headroom, suggesting they need better cost‑awareness heuristics.
- Self‑review loops are missing – Future agents should incorporate quality‑assessment steps before final assembly.
- Token cost can dominate total spend – Especially for Claude, where token usage contributed up to $25 of a $73 run.
Reproducing the Experiment
The full harness, transcripts, and video outputs are available:
- Code:
github.com/hershalb/music-video-arena - Transcripts:
- Final videos (full‑length MP4s) are embedded in the original blog post.
Final Assessment
While the autonomous agents succeeded in producing technically complete videos within the budget, the artistic quality remains far from human‑produced standards. The experiment highlights current gaps in narrative coherence, tempo synchronization, and iterative self‑editing. Nonetheless, the differing tool‑selection strategies between Claude Fable 5 and GPT‑5.6 Sol provide valuable data for future research on cost‑aware, multi‑model orchestration in long‑horizon generative tasks.