LLM Coding Benchmark

What it is

• A research‑grade benchmark that measures how well large language models (LLMs) can autonomously write, build, and validate a small Ruby on Rails application.
• The suite runs each model through two phases using the opencode tool: (1) generate the Rails code, (2) boot the app locally (Docker build + docker compose up).
• Results are stored per‑model under results/… and summarised in Markdown reports.

Key components

• Configuration – config/models.json lists every model (provider, slug, optional context overrides). The benchmark creates a temporary opencode config (config/opencode.benchmark.json) so your global settings stay untouched.
• Prompts – prompts/benchmark_prompt.txt (phase‑1) and prompts/benchmark_followup_prompt.txt (phase‑2) are the same for every run, ensuring a consistent task description.
• Scripts
  • scripts/warmup_ollama_models.py checks that locally‑hosted Ollama models load and reports the largest usable context window.
  • scripts/run_benchmark.py drives the whole experiment: prepares the config, invokes opencode run --agent build --format json for each model, saves raw logs, normalises them to result.json, and regenerates the Markdown reports.
  • scripts/analyze_results_runtime.py (with scripts/browser_probe.mjs) tries to start the generated Rails project, run Docker, and even open a headless browser to verify the app works.
• Documentation – a large set of hand‑written analyses (docs/*.md) explains the scoring rubric (0‑100 across eight dimensions), per‑model findings, multi‑agent experiments, and cost‑quality trade‑offs.

How it works

1. Warm‑up – run the Python warm‑up script to discover each Ollama model’s max context size.
2. Generate benchmark config – run_benchmark.py builds a minimal opencode config that only includes the models you want to test.
3. Execute – the script launches opencode run for each model. The tool’s built‑in system prompts differ by model family (Claude, GPT/Codex, Gemini, Kimi, etc.), a fact the repo documents as a known confound.
4. Two‑phase run
   • Phase 1: the model writes the Rails code.
   • Phase 2: the same session continues and validates the app by booting it locally and with Docker.
5. Post‑processing – logs are normalised, runtime validation is attempted, and the consolidated tables (docs/report.md, docs/report.nvidia.md) are regenerated.

What you get

• Per‑model artifacts – source code, raw opencode logs, and a result.json summarising success/failure, runtime, and cost.
• Aggregated reports – Markdown tables ranking models by overall score, cost per run, and wall‑clock time.
• Deep code reviews – the docs/success_report*.md files contain a rubric‑driven audit of each model’s output, highlighting API hallucinations, test quality, architecture decisions, and production‑readiness.
• Insights on multi‑agent setups – experiments where a planner model delegates work to a cheaper executor, with forced‑delegation variants and analysis of when orchestration helps or hurts.

Who might use it

• Researchers comparing the coding abilities of emerging LLMs (Claude Opus, DeepSeek, Kimi, Gemini, etc.).
• Teams evaluating cost‑effective options for autonomous code generation in Ruby/Rails projects.
• Anyone interested in reproducible LLM benchmarking pipelines that include real‑world build and Docker validation steps.

Limitations noted by the authors

• System prompts are injected by opencode per model family, so cross‑family scores mix model quality with harness tuning.
• Some runs depend on external services (OpenRouter, deepclaude shim) and on the availability of an Anthropic subscription for certain variants.
• The benchmark focuses on a single, relatively small Rails task; results may not generalise to larger codebases.

---

All details are taken directly from the repository’s README; no additional features have been inferred.