Shippy: Architecture and Lessons in Building High-Stakes Maritime AI Agents

Shippy: Architecture and Lessons in Building High-Stakes Maritime AI Agents

TL;DR

Hugging Face and Ai2 have introduced Shippy, a maritime AI agent designed for high-stakes operational domain awareness. To ensure reliability in environments where incorrect data can lead to wasted resources or safety risks, Shippy utilizes a modular architecture separating persona from capability, employs deterministic CLI tools to interface with live data, and operates within isolated user sandboxes.

Agent Anatomy: Soul, Skills, and Config

Shippy is structured as three distinct components to ensure versioning, auditability, and flexibility:

  • Soul: The system prompt that defines the agent's persona and behavioral boundaries. This includes explicit constraints, such as refusing to make legal determinations or speculate beyond available data.
  • Skills: Defined as plain markdown files with structured frontmatter (following the agent-skills spec), skills provide instructions for specific tasks. Current capabilities include querying the Skylight API for vessel data and events, looking up Exclusive Economic Zones (EEZ) and Marine Protected Area (MPA) boundaries, interpreting vessel track data, and generating interactive map links.
  • Config: Runtime settings that determine the agent harness (OpenClaw) and the LLM (currently Claude Opus 4.6).

These components are packaged into a versioned Docker image, allowing developers to swap models or harnesses via configuration changes without rebuilding the entire agent.

Deterministic Tools for Nondeterministic Agents

To mitigate the nondeterminism of LLMs, Shippy does not issue raw API calls. Instead, it interacts with the Skylight platform through a purpose-built, self-documenting CLI.

The CLI Layer

Direct API calls often resulted in malformed pagination, geometry encoding errors, and incorrect filter types. The Skylight CLI collapses this complexity into typed filter flags (e.g., skylight events search), handling authentication and pagination internally. To avoid pipe buffer limits and ensure programmatic access, the CLI writes all output to local JSON files on disk.

The API Layer

Underneath the CLI is a standardized API with typed schemas and field-level descriptions. This layered approach—Typed API $\rightarrow$ Deterministic CLI $\rightarrow$ Agent Skills—allows each component to be tested independently, narrowing the margin for error at each stage.

Sandboxed Hosting and Isolation via Mothership

Because Shippy serves government agencies and NGOs across 70 countries, strict data isolation is mandatory. Ai2 developed Mothership, an agent hosting platform that provisions a dedicated, ephemeral Kubernetes deployment for every user session.

  • Session Isolation: Each user session spins up a set of pods containing the agent runtime, skills, and the Skylight CLI.
  • Security: The user's Skylight JWT is injected at provision time to scope API calls to their specific data. Network access is restricted to necessary services, and files written during analysis are deleted when the session ends.
  • Capabilities: Within this sandbox, the agent can install dependencies, run code, and pull datasets for multi-step analysis without risking cross-user data leakage.

Evaluating Agent Performance over Model Performance

Rather than relying on static general-purpose benchmarks, Ai2 evaluates Shippy as a complete system (model, skills, and sandbox) against live data using the Harbor evaluation framework.

The Evaluation Pipeline

Subject-matter experts create scenarios and rubrics with weighted criteria. For example, a fishing-events query prioritizes data accuracy and boundary resolution over response style. The process follows a specific pipeline:

  1. A natural-language prompt is executed in the sandbox.
  2. An LLM judge grades each criterion from 0 to 1 with written reasoning.
  3. A weighted aggregate is compared against a fixed pass threshold.

Findings and Iteration

Recent evaluations identified specific failure modes, such as the agent overstepping into tactical recommendations rather than decision support, boundary simplification causing missed events in geometry-sensitive queries, and the hallucination of non-existent CLI commands. These results directly drive the next iteration of skill improvements.

Future Roadmap

Ai2 is expanding Shippy's capabilities with three primary focuses:

  • Agent-driven UI control: Enabling Shippy to directly manipulate the Skylight map (adjusting filters and time ranges) rather than just providing links.
  • Model routing: Implementing a system to route simple lookups to smaller, faster models while reserving frontier models for complex investigations.
  • Cross-thread memory: Developing persistent memory so the agent remembers user-specific context (such as an analyst's jurisdiction) across different conversation threads.

Sources