Prism-Shadow/penguin-harness

PenguinHarness – A Unified Framework for Agent Self‑Evolution (RSI)

What it is

• An open‑source platform that lets you run, evaluate, and automatically improve AI agents (the harness around a fixed model). It focuses on recursive self‑improvement (RSI): the harness—prompts, tools, skills, and hooks—evolves while the underlying language model stays unchanged.
• Available as a desktop app, a web app, a CLI, and an SDK, so you can work locally, on a server, or inside Docker.

Core ideas

1. RSI algorithms as plug‑ins – The framework ships with several research‑grade self‑evolution algorithms:
   • default-rsi-toolkit – simple trial‑reflect‑improve loop.
   • opro-rsi-toolkit – implements OPRO (optimizing prompts from past scored runs).
   • ape-rsi-toolkit – APE (inducing candidate instructions from examples).
   • ace-rsi-toolkit – ACE (learning a playbook from trace lessons).
   • awm-rsi-toolkit – AWM (inducing workflows from successful runs).
2. Benchmarks & scoreboard – Every run is scored on a set of built‑in benchmarks (data‑analysis, coding, automation, scientific terminal tasks, etc.). Scores are stored in an Evaluation Center and visualised as version‑over‑version charts.
3. One‑sentence app generation – Describe an agent’s purpose in a single sentence; the framework automatically creates a full‑stack app (UI, backend, run instructions) for you.
4. Cheap, model‑agnostic operation – Designed for open‑source and commercial LLMs (DeepSeek, Kimi, GLM, Hunyuan, Qwen, GPT‑6, Gemini, Claude, Inkling, etc.). The toolset is deliberately small, keeping token usage low (the README cites a data‑analysis task costing ~1/70 of Claude Code).

Key capabilities

• Browser automation via a dedicated Chrome extension or the built‑in desktop browser; agents can read pages as text and act with short scripts.
• Hot‑reloaded workbenches – each agent can generate its own UI page and server code on the fly.
• Remote‑machine execution – attach SSH‑reachable machines; conversations run on those machines inside the same UI.
• Unified Agent API – HTTP endpoint that streams AMSP events, letting any external app talk to an agent.
• Skill creation & optimisation – agents can write and refine their own reusable “Skills” (e.g., data‑analysis, code generation, human‑in‑the‑loop tools).

Supported models (any OpenAI‑compatible endpoint works, plus built‑in presets):

• DeepSeek V4, Kimi K3, GLM 5.3, Hunyuan 3, Qwen 3.8 Max, GPT 6.1, Gemini 3.8 Flash, Claude 5.5, Inkling, and many others via OpenRouter, Fireworks AI, SiliconFlow, TokenDance, etc.

Installation options

Method	Command / Link
Desktop app	Download from penguin.ooo/download (macOS dmg, Windows exe, Linux AppImage/deb)
CLI / server	npm install -g @prismshadow/penguin-cli (Node ≥ 24) or platform‑specific script (`curl …
Docker	docker run -d -p 127.0.0.1:7364:7364 -v penguin-data:/data hiyouga/penguinharness:latest
Chrome extension	PenguinHarness Browser from the Chrome Web Store

Typical workflow

# 1. Choose a model
penguin config model add --provider deepseek --model-id deepseek-flash --api-key sk-… --set-default

# 2. Run a one‑shot task
penguin run -m "Create hello.txt containing Hello, Penguin"

# 3. Start an interactive REPL
penguin chat

# 4. Launch the web UI (or desktop app) for visual benchmarking and app building
penguin web   # opens http://127.0.0.1:7364

In code (TypeScript):

import { createAgent, userText, isCompleteModelMessage } from "@prismshadow/penguin-core";

const agent = await createAgent({ agentId: "default_agent" });
const session = await agent.createSession({ workspaceDir: process.cwd() });

for await (const out of session.run([userText("Create hello.txt containing hi")], {
  approve: async () => "allow",
})) {
  if (isCompleteModelMessage(out) && out.payload.type === "text") {
    console.log(out.payload.text);
  }
}

Use cases demonstrated in the README

• Legal‑review agent – users upload contracts, the agent analyses risk clauses, explains them, and produces a revision report (cost ≈ $0.04 for five sample reviews).
• Decision‑making agent – runs a benchmark, revises its own method, and improves its score from 84 → 100 in a single iteration.
• Data‑analysis suite – achieves the best accuracy among comparable tools at a fraction of the cost of Claude Code.

Roadmap highlights

• ✅ Public benchmark suite released
• ✅ Desktop and Windows support
• ✅ Agent‑company templates
• ⬜ Company‑level self‑evolution
• ⬜ OpenShell integration (permission‑governed shell)
• More plugins and workflow extensions planned.

Community & resources

• Docs: penguin.ooo/docs
• Blog tutorials (browser automation, workbench creation, remote machines, API usage)
• Discord, X (Twitter), WeChat groups for real‑time help
• Open‑source benchmark repo: penguin-harness-benchmark

License – Apache‑2.0 (2026 Prism Shadow). Contributions are welcomed via the usual pull‑request workflow; a CONTRIBUTING.md outlines quality gates and workspace layout.

---

Bottom line: PenguinHarness is a full‑stack, model‑agnostic framework that lets developers and researchers run LLM‑based agents, evaluate them on a suite of tasks, and automatically improve the agents’ prompts, tools, and workflows using state‑of‑the‑art RSI algorithms—all from a single, cross‑platform interface.