lechmazur/writing

What it solves

This project provides a benchmark to evaluate the creative story-writing capabilities of Large Language Models (LLMs). It moves beyond simple free-form writing by testing how well models can construct coherent, original stories while adhering to strict, constrained creative briefs.

How it works

Models are given constrained briefs containing ten required elements (such as character, object, concept, setting, and tone). The benchmark then uses a pairwise comparison method:

1. Story Generation: Models write stories based on the same fixed briefs.
2. Pairwise Evaluation: Separate evaluator models read matched pairs of stories and choose which is better.
3. Order Neutralization: Stories are presented in both possible orders to prevent position bias.
4. Scoring: Individual judgments are combined into a relative comparison score and a win-chance percentage, rather than using absolute grades.
5. Validation: The system uses bridge validation to combine evidence across different versions of evaluator models.

Who it’s for

• AI researchers and developers looking to measure the creative writing quality of LLMs.
• Users interested in comparing the narrative performance of different frontier models.

Highlights

• Constrained Creativity: Measures integration of specific plot elements rather than just keyword inclusion.
• Relative Ranking: Uses a head-to-head comparison system to create a leaderboard of model performance.
• Comprehensive Data: Includes public artifacts such as story prompts, generated texts, and machine-readable leaderboard tables.
• Diagnostic Tools: Tracks evaluator agreement and word count compliance to ensure benchmark reliability.