ninjahawk/livenerf
Benchmark for tracking model capability after release.
What it solves
Claims that AI models get quietly worse after launch are hard to verify because no one has a clean day-0 baseline. livenerf provides a rigorous, deterministic-as-possible way to detect drift. It starts measuring on launch day, freezes every controllable part of the evaluation, and uses pre-registered statistics so the results aren't just vibes.
How it works
livenerf runs a small panel of 78 questions that the model sometimes gets right (calibrated to be informative). Every day it executes the same frozen prompts, pinned CLI version, and exact graders. Scores are compared over 10-day windows against the launch-week baseline. A control model runs the same questions through the same harness each day to detect if the harness or platform changed instead of the model. Output token counts are tracked as an early signal of reduced effort. All logs are append-only and every decision follows a pre-registered rule, so nobody reads tea leaves.
Who it's for
This is for anyone concerned about model performance stability after launch, including researchers, evaluators, and users of frontier models. It's also a template for anyone with a Claude Max subscription who wants to run their own continuous benchmark with minimal compute overhead (about 3.6% of the weekly plan).
Highlights
- Pre-registered protocol: decision rules are locked before data collection, and the public git timestamp proves it
- Frozen everything: pinned CLI, identical prompts, hermetic calls, no LLM judge (exact-match scoring only)
- Control arm: a separate older model catches platform-wide changes
- High statistical power: detects ~7.5 point accuracy shifts per 10-day window, plus token-output drops earlier
- Status: 52 of 60 days collected, no change detected so far (as of 2026-11-14)
- Append-only logs and canary strings for dataset provenance
Related
- Dispatch
- Project
- Dispatch
- Project
- Dispatch