MOLT Sinai Hospital Model: Five Months of Automated Bug‑Fixing as a Medical‑Team Metaphor

Takeaway

In five months, Cockroach Labs’ MOLT Sinai “hospital” pipeline added IBM Db2 migration support in two days for $4,172 in LLM tokens—164× faster and 38× cheaper than the prior 9‑month, $160 K Oracle effort—showing that a teaching‑hospital metaphor for LLM agents can produce production‑grade code at dramatically reduced cost and risk.

Why a Teaching‑Hospital Metaphor?

Conclusion: Quality, not speed, drives migration tooling; a hospital model enforces rigorous handoffs, reviews, and documentation.

  • Agents can generate code quickly, but uncontrolled output risks data corruption in a distributed SQL database.
  • Existing agent factories (e.g., Gas Town, Cursor’s SQLite rebuild) optimise for throughput, not for the safety guarantees required by database migrations.
  • A teaching hospital provides a proven framework for high‑stakes work with uneven skill levels, mandatory second opinions, and documented handoffs.

"We looked for a model with long experience managing high‑stakes work done by staff of uneven ability, with mandatory handoffs, mandatory second opinions, and literature on what goes wrong when the handoffs are sloppy. And we landed on a teaching hospital." – Rafi Rafiss, Cockroach Labs

Hospital Roles Mapped to Agent Functions

Conclusion: Each role isolates a specific responsibility, preventing agents from over‑stepping and ensuring traceability.

Role Agent Prompt / Responsibility
Triage Nurse Classifies incoming GitHub issues, admits or rejects them.
Fellow Reproduces the bug, writes a diagnostic workup, and produces a detailed treatment plan before touching code.
Review Attending A separate LLM instance that reviews the plan and code for bugs, regressions, and security issues.
Discharge Nurse Audits that the review was performed correctly, CI is green, and the commit history is clean before merge.
Chief of Medicine Human escalation point for cases the agents cannot resolve.
Supporting departments Charge Nurse (re‑runs stalled stages), Infection Control (locks pipeline on main‑branch breakage), Safety (weekly process reviews), Research (proposes new work).

Technical Implementation

Conclusion: The entire workflow is orchestrated with GitHub Actions and issue labels, making the state machine observable and reproducible.

  • State tracking: Issue labels represent the current stage (e.g., triage-in, fellow-out). Adding a label triggers the corresponding GitHub workflow.
  • Prompt loading: Each workflow loads a role‑specific prompt and writes a structured note (<!-- SINAI:TREATMENT_PLAN -->) to a chart file for downstream agents.
  • Safety rules encoded in skill files:
    • No code before a plan, no plan without review.
    • Don’t improvise – any scope change forces a new plan and review.
    • No test shortcuts – failing tests are treated as correct until proven otherwise.
    • Never flail – stuck Fellows issue an I‑PASS handoff before escalation.
    • Discharge Nurse audits the reviewer – ensures the review process was followed.
    • Precedent log – agents consult an append‑only log before escalating to the human Chief.

Production Results (April 21 – Sept 11, 2026)

Conclusion: The hospital pipeline delivered over a million lines of code with minimal human intervention and high fidelity.

Metric Value
Treatment PRs merged 1,238
Issues discharged 1,322
Agent‑generated follow‑up issues 1,299
PRs later reverted 7
Issues escalated to human Chief 92
Median PR size 527 lines added
  • The Db2 migration, the first large feature, completed in <2 days with $4,172 token cost.
  • Human‑approval mode was enabled after the first week to verify quality before shipping to customers.
  • The same pipeline subsequently produced the Migration Assistant (AI‑guided Postgres‑to‑CockroachDB migration) now in preview.

Cost Breakdown & Model Evolution

Conclusion: Token‑based costing dropped dramatically as the pipeline migrated from expensive models (Opus) to a tiered approach using Fable for planning and selective escalation.

  • Total token spend: $135,000 (≈ $84 per issue).
  • Initial stages used Claude Opus for every step; later stages switched to Fable for planning and dynamically selected models for implementation, reducing per‑issue cost.
  • Cost spikes correspond to model‑switch tuning cycles and workload mix (UI work incurs higher token usage due to image processing).

Self‑Improvement: Redundant Skill Files

Conclusion: About a quarter of the skill‑file text was redundant, inflating token usage; pruning will yield measurable savings.

  • Audit of 25 skill files (~100 k words) found 23 % removable.
  • The Discharge Nurse loads ~21,800 words (≈ 29 k tokens) per run; redundancy directly increased cost.
  • Future work includes building a linter to detect and eliminate duplicated rules.

Lessons Learned

What Worked

  • Persona framing – Short role introductions (“You are a Fellow…”) gave agents strong contextual grounding, leading to nuanced language (e.g., polite escalation messages).
  • Plan‑first review – Only <10 % of plans were rejected, but those rejections prevented downstream bugs, saving large amounts of rework.
  • Hard ceiling on change size – Any work >1,000 lines triggers automatic decomposition, keeping each PR reviewable.
  • Comprehensive audit trail – Every action (plan, I‑PASS, review verdict, token usage) is archived in a companion branch, enabling full post‑mortems.

What Needs Improvement

  • Over‑bureaucracy – Agents sometimes created duplicate‑close actions or escalated trivial UI tweaks, leading to unnecessary cycles.
  • Circuit‑breaker for endless rework – Implemented a safety review after three rework rounds; further refinement is ongoing.
  • Self‑generated backlog – Nearly 50 % of issues were auto‑created; throttling the Research Department reduced noise.
  • Learning opportunities for junior engineers – Current model favours senior‑level throughput; plans are underway to let humans craft plans and receive agent feedback for skill development.

Community Feedback (HN Highlights)

"This feels like a lovely example of high‑quality systems design… the metaphor efficiently encodes complex relationships." – finnborge

"Role‑playing with LLMs is not the secret to producing amazing code." – anonymous908213

"Why use GitHub issues instead of a dedicated ticket system?" – reachableceo

"Great post – modern medical team practices translate well to AI agents." – cs702

"The pipeline generated a lot of internal work; throttling was necessary." – kinduff

These comments underscore both enthusiasm for the metaphor and concerns about tooling choices and bureaucracy.

Next Steps

Conclusion: The team will refine cost efficiency, expand educational features for junior engineers, and explore fully‑autonomous operation for low‑risk issue classes.

  • Continue pruning skill‑file redundancy and adding a prompt‑linter.
  • Enable human‑crafted plans with agent review to create a learning loop.
  • Validate the correctness of the Db2 migration with independent testing.
  • Experiment with a post‑merge “travel‑nurse” stage for acceptance testing.
  • Open the Sinai framework for broader internal adoption (four repos already use it, more planned).

The MOLT Sinai experiment demonstrates that a well‑engineered, hospital‑style LLM workflow can dramatically accelerate complex engineering tasks while preserving the safety guarantees required for production database tooling.

Sources

Related