Human-Agent-Society/reef

Reef – Infrastructure for Continually Self‑Improving Agents

What it is – Reef (available as the reef‑infra PyPI package) is an open‑source framework that ties together four stages of a learning loop for autonomous agents:

1. Serve – runs an inference server (compatible with OpenAI/Anthropic APIs) and records every request/response.
2. Observe – matches incoming feedback (scores, textual comments, or structured data) to the recorded interactions.
3. Grow – runs a configurable recipe that turns eligible feedback into an update – either a new set of model weights (via Slime, SGLang, etc.) or a modified agent harness (prompts, rules, skills, extensions).
4. Commit – evaluates candidate updates, selects the best ones, and publishes them to a version‑controlled artifact store (using Git‑LFS), so the serving component can hot‑swap to the newest version without downtime.

Why you’d use it – If you have an agent that you want to keep getting better from real‑world usage, Reef gives you:

• A live inference endpoint that automatically logs interactions.
• A feedback API (/reef/report) that can accept simple scores or richer signals.
• Built‑in support for two “learning surfaces”:
  • Model‑weight training – requires a GPU‑capable training stack (Slime, SGLang) and a trainable model.
  • Harness evolution – works with any model endpoint and lets the agent improve its prompts, rules, or custom code (the Reefine recipe).
• Versioned artifact management so updates are reproducible and can be rolled back.

Key components

Component	Role
service/ & runtime/	Serve traffic, store interaction records, and host the current model/harness.
storage/records.py	Persists interactions and feedback.
train/processors/	Determines which records are eligible for learning.
recipe/	Declarative “recipes” that define how to turn feedback into updates (e.g., SAO, Reefine, GEPA).
train/	Batching, job orchestration, and evaluation of candidate updates.
artifact/ & surface/	Git‑LFS‑backed version history and delivery of new weights or harness artifacts.

Typical workflow

1. Start a server – reef serve --inference.model-path Qwen/Qwen2.5-1.5B-Instruct (or a harness‑evolving recipe).
2. Send a request – client calls the OpenAI‑compatible /v1/chat/completions endpoint; the response header x-reef-agent-record-id gives a receipt ID.
3. Report feedback – POST to /reef/report with score, feedback, and the receipt ID(s).
4. Reef’s recipe consumes enough feedback, runs a training step (or harness‑generation step), evaluates candidates, and publishes the chosen update.
5. Live update – the serving runtime automatically loads the new artifact; no restart needed.

Recipes & benchmarks – The repo ships a catalog of ready‑made recipes for different problem families:

• Scientific discovery (e.g., TTT‑Discover) – weight‑only training.
• Continual learning on a task stream (SAO) – can evolve both weights and harnesses (Meta‑Harness, GEPA).
• Learning from usage (OpenClaw‑RL, SkillClaw, Reefine) – harness‑only improvement. Each recipe includes code, documentation, and result pages for standard benchmarks such as IMOAnswerBench, CEO‑Bench, and simulated student streams.

Installation – Recommended with uv:

uv venv && source .venv/bin/activate
uv pip install reef-infra   # or `pip install -e .` from source

Git‑LFS must be installed for artifact storage.

Community – Active Discord, WeChat, GitHub Discussions, and an RFC‑based contribution process. The project is under the Apache‑2.0 license.

---

All details above are taken directly from the repository’s README.