Ed‑o‑meter Benchmark Shows GLM‑5.3 Beats Anthropic and OpenAI Models at One‑Fifth Cost

GLM‑5.3 dominates the Ed‑o‑meter leaderboard, delivering 100 % pass rate at roughly one‑fifth the cost of GPT‑5.5

Takeaway: In the latest Ed‑o‑meter benchmark, the open‑weight GLM‑5.3 model cleared every test corner with a perfect pass rate, a 9.3 rubric score, and a total lap cost of $0.28—about 20 % of GPT‑5.5’s $1.43 price tag. It is the recommended single‑model choice for general‑purpose workloads, while faster or cheaper alternatives exist for specific use‑cases.

Benchmark methodology is transparent and uniform across 17 models

Conclusion: The Ed‑o‑meter runs the same 28 real‑world tasks on every model using identical prompts, deterministic grading, and a single OpenRouter streaming path, so differences reflect model behavior rather than experimental variance.

  • All models were evaluated on a fixed “lap” consisting of five corners—Coding, Data, Realworld, Security, and Tool‑use—each with multiple unit‑test style tasks.
  • Pass rates are reported with Wilson 95 % confidence intervals because each model was run only once per task.
  • Latency is measured as median time‑to‑first‑token (TTFT) under the same streaming conditions.
  • Costs are calculated from the provider‑quoted per‑token rates applied only to answering trials (refusals are logged separately).
  • The LLM rubric (quality score) is generated by a separate model (fable‑5) on saved answer text; its self‑bias is disclosed in footnotes.
  • The entire harness, task definitions, and binary checkers are open‑source under the MIT‑licensed Featherbench repository.

Overall results place GLM‑5.3 at the top of both quality and cost efficiency

Conclusion: GLM‑5.3 is the only model that achieved a 100 % pass rate across all five corners, scoring a 9.3 rubric and costing $0.28 per full lap.

Rank Model Pass Rate (95 % CI) Rubric Security Pass TTFT Lap Cost Cost / Task
1 glm‑5.3 100 % [88–100] 9.3 100 % 16.3 s $0.28 $0.0101
2 deepseek‑v4‑pro 96 % [82–99] 8.7 83 % 40.0 s $0.081 $0.0029
3 gemini‑3.6‑flash 96 % [82–99] 8.8 83 % 6.6 s $0.48 $0.0170
4 gpt‑5.5 96 % [82–99] 8.7 100 % 13.2 s $1.43 $0.0510
… … … … … … … …

The table shows the top four models; the full leaderboard includes 17 models.

Model‑specific highlights

GLM‑5.3 is the all‑round champion

Conclusion: GLM‑5.3 clears every corner (Coding, Data, Realworld, Security, Tool‑use) with 100 % pass, making it the safest single‑model pick despite a modest 16.3 s TTFT.

  • Its rubric of 9.3 is the third‑highest among all models.
  • Cost per task ($0.0101) is dramatically lower than any closed‑source competitor.
  • The slower response time is the main trade‑off; for latency‑critical applications, GPT‑5.5 is faster (13.2 s) with comparable security.

GPT‑5.5 offers the best speed‑to‑quality ratio

Conclusion: GPT‑5.5 matches GLM‑5.3’s 100 % security pass and 89 % real‑world pass while delivering a 13.2 s TTFT, but its lap cost is $1.43, roughly five times higher.

GPT‑5.6‑luna is the cheapest workhorse for high‑volume, low‑risk jobs

Conclusion: At $0.064 per lap ($0.0023 per task) and a 5.3 s TTFT, GPT‑5.6‑luna is ideal for batch workloads where occasional failures are acceptable.

  • Its overall pass rate is 79 % and security pass drops to 33 %, so results must be validated before production use.

Haiku‑4‑5 delivers ultra‑low latency at modest cost

Conclusion: Haiku‑4‑5 answers in 0.9 s, costs $0.12 per lap, and achieves a 96 % overall pass, making it the fastest model for interactive use cases.

Kimi‑K3 achieves the highest rubric score (9.5)

Conclusion: Kimi‑K3’s rubric of 9.5 indicates the best judged answer quality, but its 26.4 s TTFT and 75 % data‑corner pass limit its suitability for real‑time applications.

Opus‑5 shows a measurement artifact due to provider‑side filtering

Conclusion: Opus‑5’s 43 % coding pass is not a true capability deficit; four benign coding‑debug‑* tasks were blocked before any token was generated, inflating the failure count.

Safety findings expose weaknesses in the GPT‑5.6 family

Conclusion: The GPT‑5.6 trio (luna, terra, sol) exhibits low security pass rates (33 %–50 %) because they triggered jailbreak canaries in 11 of 12 security cells.

  • By contrast, GPT‑5.5, Claude’s Haiku series, and the Opus‑5 model achieved 100 % security pass.
  • Users should employ robust red‑teaming and external filters when deploying GPT‑5.6 variants.

Community reactions surface skepticism and praise

Conclusion: Hacker News commenters are split between admiration for the benchmark’s transparency and doubts about its validity, especially regarding the high scores for Haiku and the reliance on a self‑judging rubric.

"This whole thing immediately reads as Claude generated, making it hard to take seriously. Why do these results contradict existing serious attempts at benchmarking LLMs?" – hellohello2

"Almost 10 models are passing the benchmark >95% – isn’t that substantially overly saturated? I'm also really skeptical of benchmarks that place any Haiku model very high." — jchw

"One problem is that $30 per run is really noisy for many verifiable tasks. We just published GLM‑5.3 results on our multi‑agent coding evaluations and it's definitely an impressive model, coming in around #6. For the price, it's actually not Pareto optimal, falling slightly behind Grok‑4.6 (which is faster and the same price)." — gertlabs

"IDK I use open models every day for personal projects, and closed models for work. Open models are all decidedly far behind Fable and a good bit behind Opus as well. All of these posts read like motivated/wishful thinking to me." — solenoid0937

"If you run one model, run glm‑5.3. That is a horrible take‑away from this, with only 28 tasks and a high pass rate for most models, it says almost nothing. Test a model for your use case and use the fastest, smallest, cheapest model that 100 % satisfies your use case." — CMay

These comments highlight three recurring concerns:

  1. Benchmark breadth – 28 tasks may not capture all real‑world complexities.
  2. Rubric bias – The quality score is generated by fable‑5, which self‑judges and may over‑rate its own outputs.
  3. Provider filtering effects – Blocked tasks (e.g., Opus‑5 coding‑debug) can artificially depress pass rates.

Limitations and caveats disclosed by the authors

Conclusion: The Ed‑o‑meter authors acknowledge four key sources of potential distortion.

  1. Rubric scoring is performed retroactively by fable‑5; self‑judgment is flagged in the table.
  2. Fable‑5’s own rubric (9.3) is based on a limited 11‑trial sample and may be biased upward.
  3. Recipe‑checker false‑positives caused three models to be marked as passing a vegetarian recipe task despite a non‑ingredient mention.
  4. Cost calculations for fable‑5 and opus‑5 exclude refusals, which would otherwise make those models appear cheaper.

Running the Ed‑o‑meter yourself

Conclusion: The benchmark is fully reproducible; anyone can clone the Featherbench repository and evaluate additional models.

  • The harness, task definitions, and binary checkers are available at https://github.com/ed‑is‑ai/featherbench (MIT license).
  • To request a new model, open a GitHub issue with the model name, API route, and justification.
  • The entire 28‑task suite costs roughly $30 to run on the current price schedule, making it affordable for small‑scale research.

Bottom line

Conclusion: GLM‑5.3 currently offers the best combination of universal correctness, reasonable latency, and ultra‑low cost, positioning it as the default choice for developers who need a single, reliable LLM. Faster alternatives (GPT‑5.5, Haiku‑4‑5) and cheaper batch‑oriented models (GPT‑5.6‑luna) fill niche roles, while the security shortcomings of the GPT‑5.6 family and the filtering artifacts affecting Opus‑5 remind practitioners to validate model behavior against their own safety requirements.

Key numbers at a glance

Metric GLM‑5.3 GPT‑5.5 GPT‑5.6‑luna Haiku‑4‑5
Overall pass 100 % 96 % 79 % 96 %
Security pass 100 % 100 % 33 % 100 %
Rubric 9.3 8.7 8.6 (self‑judged) 7.4
TTFT 16.3 s 13.2 s 5.3 s 0.9 s
Lap cost $0.28 $1.43 $0.064 $0.12
Cost / task $0.0101 $0.0510 $0.0023 $0.0044

Future work

  • Expanding the task suite beyond 28 items will reduce statistical noise and improve confidence in pass‑rate differences.
  • Introducing multiple runs per model would tighten Wilson intervals and expose variability.
  • Re‑judging the rubric with an independent, non‑self‑biased evaluator would address concerns about fable‑5’s self‑scoring.
  • Tracking TTFT across different routing providers could surface network‑level latency effects.

Takeaway for practitioners: Start with GLM‑5.3 for general workloads, switch to GPT‑5.5 when latency matters, and reserve GPT‑5.6‑luna for cheap, high‑throughput pipelines that can tolerate security‑related failures. Always validate the model against your own domain‑specific tasks before committing to a production deployment.

Sources

Related