hud-evals/hud-python

What it solves

HUD provides a unified platform for building, deploying, and scaling reinforcement learning (RL) environments for AI agents. It solves the fragmentation between environment definitions and the agents that run within them, allowing developers to create tasks across diverse domains—such as coding, web browsing, computer-use, and robotics—and use those same tasks for both evaluation (evals) and model training.

How it works

HUD uses a "protocol-first" approach where the environment and the agent communicate via a thin envelope. The environment provides a manifest of its capabilities, a prompt to start a task, and a reward (grade) upon completion.

Between these steps, the agent interacts directly with the environment's capabilities, which include:

• ssh: Sandboxed shell and file access.
• mcp: Tools via the Model Context Protocol.
• cdp: Browser control via Chrome DevTools Protocol.
• rfb: Full computer-use via VNC (screen, keyboard, mouse).
• robot: Schema-driven observation/action loops over WebSockets.

Developers define tasks using async generators (templates) that yield prompts and receive rewards. These environments can be deployed as images and run remotely or locally against various models (Claude, OpenAI, Gemini) using built-in harnesses.

Who it’s for

It is designed for AI researchers and developers building autonomous agents who need a standardized way to benchmark their models across different modalities and use the resulting reward signals to train models via RL loops (e.g., GRPO or PPO).

Highlights

• Cross-Domain Support: Unified interface for coding, browser, GUI, and robotics environments.
• Protocol-First Design: Decouples the environment from the agent harness, ensuring environments outlive specific models.
• Integrated Training Loop: Converts evaluation traces and rewards directly into training data for RL.
• Remote Deployment: Ability to build, deploy, and sync tasksets to run at scale on the HUD platform.
• Broad Model Compatibility: Native support for major LLM providers and the ability to bring custom agent harnesses.