SakanaAI/AI-Scientist-v3

The Agentic Scientist – Automated AI‑driven scientific discovery

What it is

• An open‑source framework that treats large‑language‑model coding agents as virtual scientists. Starting from a plain task.md description, the system orchestrates a multi‑stage pipeline that:
  1. Drafts a formal task definition and evaluation rubric.
  2. Investigates – runs an idea‑>experiment‑>evaluation loop where agents write code, execute experiments, and keep the best program.
  3. Ablates – compares the winning program against baselines and deliberately weakened variants.
  4. Final‑evaluates – runs a sealed, reproducible assessment of the final solution.
  5. Post‑hoc analyses – mines logs, metrics, and visualisations for extra evidence.
  6. Writes a paper – assembles a manuscript (PDF) that cites the generated evidence, optionally sending it to an AI reviewer.
• All stages are reproducible: each run creates a deterministic directory tree under results/ that stores logs, artefacts, and the final paper.

Key concepts

• Coding agents – the framework can drive Claude, OpenAI Codex, Gemini, Cursor, Opencode, and other LLM‑based code generators via the Headless CLI.
• Docker‑isolated phases – every pipeline stage runs in its own container, keeping the host safe from the agents’ unrestricted command execution.
• Hydra configuration – all knobs are exposed as key=value overrides, making it easy to change agents, models, budgets, GPU counts, etc.
• Presets & examples – ready‑made tasks (e.g., a CPU‑only circle‑packing problem, a GPU‑heavy GRPO benchmark) live in examples/ with matching config presets under configs/preset/.

Installation

1. Install Docker.
2. Build the provided agent image:

   docker build -t agentic-scientist/agent-cli:latest -f src/docker/agent-cli/Dockerfile .
   

3. Install the Python environment (requires Python ≥ 3.11 and uv):

   curl -LsSf https://astral.sh/uv/install.sh | sh
   uv sync   # creates .venv and installs the `agsci` CLI
   source .venv/bin/activate
   

4. Create a .env file at the repository root containing the API keys for the LLM(s) you plan to use (e.g., ANTHROPIC_API_KEY, OPENAI_API_KEY). The file is mounted into the containers at runtime – never commit it.

Quick start

# Run the default CPU‑only circle‑packing example
agsci run examples/circpack

• Use --dry-run to see the exact commands without executing.
• Override any setting, e.g. limit the investigation budget:

  agsci run examples/circpack investigation.execution.max_idea_cycles=2
  

• After the run finishes, the generated paper is at: results/circpack/claude/<run_id>/paper/paper.pdf
• Inspect the whole run in a local web UI:

  agsci inspect start   # opens http://127.0.0.1:8765
  

Typical workflows

• Reproducing a published campaign – use a preset (preset=grpo) that encodes the original budgets, GPU requirements, and paper format.
• Swapping agents/models – change agent.name and agent.model on the command line to compare Claude, Opencode, Bedrock‑backed Claude, etc.
• Resuming or partial runs – point agsci run at an existing run directory and specify execution.phases= to continue only the missing stages.
• Standalone investigation loop – run just the idea‑experiment loop (useful for rapid prototyping):

  agsci investigate agent.name=claude paths.task_dir=examples/circpack execution.workers=2 execution.max_idea_cycles=4
  

• AI‑reviewed manuscript – add the review phase to send the PDF to an OpenAI‑backed reviewer and obtain a JSON review.

Safety notes

• Agents are launched with full system permissions (--allow yolo). They can execute arbitrary commands and see any API keys you mount. Run everything inside the Docker image and only feed trusted task files.
• The agsci doctor command checks that your host, credentials, Docker image, and GPU setup satisfy the requirements before a run starts.

Documentation

• Installation & Docker details – docs/install.md
• Full pipeline description – docs/pipeline.md
• Investigation loop usage – docs/usage.md
• Task drafting (experimental) – docs/task_drafting.md
• Additional pipelines (variants, post‑hoc) and utilities are linked from the README.

License

• Apache 2.0 (see LICENSE).

Citation If you use the framework in research, cite the accompanying arXiv paper:

@article{agentic_scientist,
  title   = {The Agentic Scientist: Orchestrating Coding Agents for Automated Scientific (Co-)Discovery},
  author  = {Lange, Robert Tjarko and Grillotti, Luca and Yamada, Yutaro and Charakorn, Rujikorn and Ha, David},
  journal = {arXiv preprint arXiv:2610.18933},
  year    = {2026}
}