akitaonrails/llm-coding-benchmark

What it solves

This project provides a standardized framework for benchmarking the autonomous coding capabilities of Large Language Models (LLMs). It specifically tests models' ability to build a fixed Ruby on Rails application based on a brief, evaluating them on technical correctness, API adherence, and production-readiness rather than simple file or test counts.

How it works

The benchmark runner uses the opencode tool to execute coding runs across a variety of local (Ollama) and cloud models. The process typically involves two phases: an initial build phase and a validation phase that checks for local boot, Docker build, and Docker Compose functionality. Results are then analyzed using a 0-100 holistic rubric across eight dimensions, including deliverables, error handling, and architecture. The project also includes tools for warming up local models to verify context window limits and a runtime validator to probe generated projects.

Who it’s for

It is designed for AI researchers, developers, and LLM evaluators who want to compare the real-world coding performance, cost, and speed of different frontier models and agentic harnesses (such as Claude Code or Codex) using a concrete software engineering task.

Highlights

• Multi-Harness Testing: Compares performance across different execution environments like opencode, Claude Code, and Codex.
• Rigorous Rubric: Uses a detailed 8-dimension audit to identify "benchmark-killing" patterns like hallucinated APIs and tests that mock those hallucinations.
• Hardware Profiles: Supports separate configurations for different hardware setups (e.g., AMD server vs. NVIDIA workstation) to manage VRAM and context windows.
• Orchestration Analysis: Evaluates whether multi-agent planner/executor patterns actually improve quality or just increase cost and time.
• Shim Integration: Utilizes the deepclaude shim to enable benchmarking of models like DeepSeek V4 Pro within the Claude Code autonomous loop.