lllllllama/RigorPilot-Skills

RigorPilot Skills – Reproducing AI‑research READMEs with Auditable Evidence

What it is – RigorPilot Skills is a collection of agent‑compatible “skills” that let an AI‑assistant (or a human using the skills CLI) automatically run the commands documented in a research repository’s README, capture the exact output, and insert a concise evidence card back into the README. The original file is left untouched; the tool writes a side‑by‑side annotated copy (RIGORPILOT_README.md) that shows, for each section, whether the command succeeded, was blocked, or needs manual approval.

Why it matters – Reproducing results from papers is notoriously fragile. RigorPilot records the process (commands, logs, resource usage, any deviations) and stores it in a structured bundle (repro_outputs/). This makes it possible to audit whether a claim in a paper truly matches what was executed, without rewriting the original documentation.

---

Quick‑start (install & run)

# Requires Node.js/npm (the CLI is a small npm package)
# Install every skill (covers reproduction, analysis, safe debugging, etc.)
npx skills add lllllllama/rigorpilot-skills --all

# Or install only the core reproduction skill
npx skills add lllllllama/rigorpilot-skills --skill ai-research-reproduction

1. Open the target repository in an agent that understands the Skills protocol (e.g., a LangChain‑based assistant).
2. Ask the agent to run the ai-research-reproduction skill, pointing it at the repository’s README.
3. The skill executes the smallest documented evaluation, writes all logs to repro_outputs/, and produces an annotated README beside the original.

---

Core capabilities (the “Skill” index)

Goal	Skill name	What it does
Reproduce a paper’s experiment from its README	ai-research-reproduction	Parses commands, runs them in a bounded environment, records logs, and inserts per‑section evidence cards.
Pure repository inspection (no execution)	analyze-project	Gathers metadata, dependency lists, and static code metrics.
Bootstrap data, weights, and environment	env-and-assets-bootstrap	Downloads required datasets or model checkpoints in a controlled way.
Run a single documented inference/evaluation	minimal-run-and-audit	Executes a command, captures stdout/stderr, and produces a minimal evidence card.
Conservative training runs	run-train	Starts a short, resource‑capped training loop and records loss curves.
Safe debugging before any code change	safe-debug	Runs static analysis and sandboxed test runs, flagging blockers.
Coordinated exploratory experiments	ai-research-explore	Manages a “candidate” branch, records decisions, and keeps exploration separate from trusted baseline.
Apply a candidate change on an isolated branch	explore-code	Checks out a fresh branch, applies a patch, and prepares it for execution.
Execute a bounded candidate experiment	explore-run	Runs the patched code under the same evidence‑capture regime.

---

Evidence bundle (what you get after a run)

• repro_outputs/ANNOTATED_README.md – the original README with a small badge‑style verdict after each heading.
• SUMMARY.md, COMMANDS.md, LOG.md, status.json – human‑readable summary plus machine‑readable status.
• PATCHES.md, SCIENTIFIC_CHANGELOG.md, COMPARABILITY_REPORT.md – optional files describing any code changes and how they affect scientific claims.
• _runtime/<run_id>/ – raw process snapshots, resource samples, and stdout/stderr for forensic inspection.
• agent_state.json, trajectory.jsonl – if a language‑model runner is used, these contain the model’s tool‑call history and usage metrics.

---

Validation & reliability

• Local regression suite: python scripts/run_all_tests.py reports 69/69 passing on the repo’s own self‑tests (as of 2026‑09‑07). CI badges show Windows, Linux, and macOS passes.
• External case studies: The README lists four reproducible case studies (micrograd, minGPT, PyTorch‑MNIST, nanoGPT‑Shakespeare). Each shows the original README preserved byte‑for‑byte with section‑level evidence cards. All four external protocols passed the deterministic benchmark suite.
• Bounded execution: The tool runs commands in the host environment but enforces explicit user authorisation for any large download or long‑running training. No OS‑level sandbox is provided; users must trust the target repository.
• Model‑loop status: An optional Anthropic‑based model runner is included, but the README notes that no live‑model runs have succeeded yet (three attempts returned HTTP 502). The runner is therefore experimental and not required for the core reproducibility workflow.

---

Limitations & open work

• No OS sandbox – commands can access the host filesystem and network; security relies on the user’s trust in the source repo.
• Model‑runner not yet functional – the “standalone model runner” has no successful live‑model acceptance recorded.
• Partial cases – some showcased repositories (e.g., minGPT) are marked selection‑only or partial; they demonstrate the workflow but do not complete full training or achieve paper‑reported scores.
• Resource quotas are soft – the system checks budgets but does not enforce hard OS limits; large GPU jobs must be manually guarded.
• Exploration results are never auto‑promoted – candidate experiments remain separate from the trusted baseline unless a human explicitly merges them.

---

Who might use it

• AI‑research reproducibility auditors who need a repeatable, auditable pipeline for verifying claims in papers.
• LLM‑powered research assistants that can invoke the Skills API to automatically run and document experiments.
• Developers of reproducibility platforms looking for a standard “evidence‑card” format that can be overlaid on existing READMEs.

---

License & community

• MIT‑licensed, open‑source.
• Contribution guidelines, security policy, and an engineering roadmap are provided in the repo.
• The project is listed on the Skillselion leaderboard and integrates with the skills.sh ecosystem.

---

Bottom line – RigorPilot Skills is a genuine software project that provides a structured, auditable way to execute the commands a research paper documents, preserving the original README while adding machine‑verified evidence. It is aimed at improving rigor in AI‑research reproduction, and it ships a set of reusable “skills” that can be called from LLM‑agents or the command line.