lechmazur/writing

This benchmark tests how well LLMs incorporate a set of 10 mandatory story elements (characters, objects, core concepts, attributes, motivations, etc.) in a short creative story

lechmazur/writing – LLM Creative Story‑Writing Benchmark

What it is

  • A public benchmark that evaluates large language models (LLMs) on their ability to write short stories under tightly‑specified creative briefs.
  • Each brief requires the story to incorporate ten specific elements (character, object, concept, …, tone) and stay within a 600‑800 word length.

How it works

  1. Prompt generation – A pool of combinatorial briefs is screened for coherence and originality. The strongest brief for each seed is fixed for all writer models.
  2. Story generation – Every LLM under test writes a story to the same brief.
  3. Pairwise comparison – For each brief, two stories are presented to an evaluator LLM in both possible orders. The evaluator chooses the better story.
  4. Aggregation – Choices from many evaluator models (both older and newer) are combined using a Thurstone statistical model, correcting for presentation order. The result is a relative score for each writer model, centered at zero.
  5. Uncertainty – Scores are bootstrapped (300 resamples) to give 95 % confidence intervals and estimated win‑chance percentages.

What’s included

  • Data: prompts (prompts_wc/), generated stories (stories_wc/<model>/), raw comparison matrices, evaluator‑agreement tables, and the full machine‑readable leaderboard (data/writing_mistral_large4_20261007/).
  • Visuals: rating heat‑maps, evaluator‑agreement matrix, word‑count compliance chart, and pairwise‑margin heat‑map.
  • Reports: qualitative pair‑analysis markdown files describing narrative habits of each model.
  • Historical archives: earlier releases (e.g., August 23 2026) with absolute 0‑10 rubric scores for reference.

Current leaderboard (excerpt)

Rank Model Score Win % 95 % CI
1 Claude Opus 5 (xhigh) 3.777 93 % 3.722‑3.839
2 Claude Fable 5.1 (high) 3.760 93 % 3.675‑3.860
… … … … …
57 Grok 4.5 (high) -5.040 2 % -5.135‑-4.956

Key findings

  • Scores are relative; a positive score means the model outperforms the average of the 57 evaluated writers.
  • Evaluator agreement varies (correlation matrix provided); high agreement does not guarantee evaluator accuracy, only consistency.
  • Word‑count compliance is tracked but does not affect quality scores.
  • Missing stories or judgments are excluded; intervals reflect only the data that were actually compared.

How to use the benchmark

  1. Clone the repo and generate stories for a new LLM using the same brief format.
  2. Follow the Method Summary steps to create paired comparisons and feed them to the existing evaluator models (or your own evaluator).
  3. Run the provided aggregation script (not detailed in the README but implied) to obtain a Thurstone score and confidence interval.
  4. Compare the new score against the published leaderboard to gauge relative creative performance.

Updates

  • The benchmark is actively maintained; new writer models are added almost weekly (e.g., Mistral Large 4 high on Oct 8 2026, Claude Opus 5.5 high on Sep 26 2026).
  • Both the data and the evaluation methodology evolve, but the core scoring pipeline (pairwise comparison → Thurstone model) remains constant.

Related work Lech Mazur’s organization also hosts many other LLM benchmarks (negotiation, deception, translation, etc.) linked at the bottom of the README.

Who might find this useful

  • Researchers comparing LLM creativity under controlled constraints.
  • Developers of new LLMs who want a quantitative, reproducible measure of story‑writing ability.
  • Anyone interested in systematic, pairwise evaluation of generative models beyond simple token‑level metrics.

Related

  • Project
    petergpt/bullshit-benchmark

    BullshitBench is an open‑source benchmark that measures whether LLMs detect and reject nonsensical prompts. It provides two question suites (55 and 100 items), a three‑judge scoring system, and an interactive dashboard for comparing model variants.

  • Project
    Tomsawyerhu/Chinese-WebNovel-Skill

    A modular, retrieval‑augmented prompt framework for AI‑assisted Chinese web‑novel writing. It splits the workflow into a short router (`SKILL.md`) and many self‑contained sub‑skills (concept planning, opening, plot logic, dialogue, consistency review, etc.). Each sub‑skill ships tutorials, runtime rules, good/bad examples, and may use local novel corpora via a provided search script. The system routes a user’s brief idea to the appropriate module, pulls relevant example excerpts, composes a concise prompt, generates text, and finally runs a consistency review. Designed for Chinese novel authors, prompt engineers, and researchers interested in modular LLM writing pipelines.

  • Project
    akitaonrails/llm-coding-benchmark

    A reproducible benchmark suite that runs LLMs (both local Ollama models and cloud providers) through a two‑phase autonomous coding task: generate a small Rails app, then build and validate it with Docker. It provides scripts for warm‑up, execution, and runtime validation, plus extensive markdown reports and hand‑written code reviews that score models on a 0‑100 rubric across eight quality dimensions.

  • Dispatch
    Detecting LLM-Generated Text with Classical Machine Learning – Results and Insights

    A scikit-learn TF‑IDF + LinearSVC pipeline can distinguish modern LLM‑generated Chinese text from human‑written text with ~85% sentence‑level accuracy, demonstrating that “classical” ML still works for AI‑text detection.