Nucleus: A Security-Hardened, Nix-Native Container Runtime

Nucleus is a minimalist, security-hardened container runtime for Linux designed for AI agents and production services. Unlike traditional runtimes, Nucleus eschews the OCI image-and-distribution model in favor of a fully declarative approach using Nix, providing isolated execution environments with near-zero overhead.

High-Performance Isolation and Cold Starts

Nucleus achieves significantly faster startup times and near-native performance by leveraging Linux kernel primitives directly.

  • Cold Start Performance: Nucleus starts containers in approximately 12 ms, compared to ~500 ms for Docker.
  • Steady-State Performance: Benchmarks using PostgreSQL 18 (pgbench) show that Nucleus maintains performance near bare-metal levels. In read-heavy SELECT-only tests, Nucleus achieved an average of 105,965 TPS with worker I/O, slightly exceeding bare-metal results (100,222 TPS), which the authors attribute to benchmark noise rather than a guaranteed speedup.

Declarative Architecture and Nix Integration

Nucleus is built specifically for the Nix ecosystem, replacing mutable image layers with reproducible Nix store closures.

Nix-Native Root Filesystems

Instead of pulling images from a registry, Nucleus uses nucleus.lib.mkRootfs to build minimal, reproducible root filesystems. These are mounted as read-only closures, ensuring that the runtime state is auditable and repeatable. For AI agents, nucleus.lib.mkAgentToolchainRootfs provides a specialized toolchain rootfs to avoid depending on mutable host binaries.

NixOS Module Support

Nucleus provides a first-class NixOS module, allowing containers to be declared as systemd services. This integration enables:

  • Automatic Lifecycle Management: Containers are managed as nucleus-<name>.service units with journald logging and sd_notify readiness.
  • Declarative Configuration: Resource limits, networking, and security policies are defined within the NixOS configuration.
  • Integrated Secrets: Support for systemd-creds and in-memory tmpfs for sensitive data.

Security Hardening and Policy Enforcement

Nucleus implements a "fail-closed" security model, particularly in its Production and Strict Agent modes.

Kernel-Level Isolation

Nucleus utilizes a comprehensive suite of Linux kernel primitives:

  • Namespaces: PID, mount, network, UTS, IPC, user, cgroup, and optional time isolation.
  • cgroups v2: Strict resource limits for CPU, memory, PIDs, and I/O.
  • Landlock LSM: Path-based filesystem access control (Linux 5.13+).
  • seccomp: Syscall whitelist filtering with trace-based profile generation.
  • Capabilities: All capabilities are dropped by default.

Service Modes

Nucleus operates in three distinct modes to balance flexibility and security:

Feature Agent Mode Strict Agent Mode Production Mode
Security Best-effort / Warn-and-continue Fail-closed / Forbidden degraded security Fail-closed / Strict invariants
Rootfs Host bind mounts (default) Host bind mounts (default) Required Nix closure (--rootfs)
Networking Optional / Host allowed Native host forbidden Native host forbidden
Cgroups Best-effort Required Required
PID 1 Direct exec Direct exec Mini-init for zombie reaping

Egress Control

In production bridge mode, Nucleus defaults to a deny-all OUTPUT policy. Access is granted only via explicit --egress-allow (CIDR) or --egress-domain (DNS name) flags, which are implemented via iptables rules within the container's network namespace.

Runtime Options and Extensibility

gVisor Integration

Nucleus provides first-class integration with gVisor (runsc), allowing users to run workloads with an application kernel for enhanced security. It generates the necessary OCI bundle and config.json for gVisor, supporting specific network modes like gvisor-host.

Multi-Container Topologies

Through nucleus compose, users can define multi-container stacks in a TOML format. This includes a dependency DAG that ensures services are started and stopped in the correct order, with reconciliation logic to align the running state with the desired configuration.

Formal Verification

To ensure reliability, Nucleus employs spec-driven development. State machines are formally verified using TLA+ and the Apalache model checker, with model-based tests in Rust to verify the implementation against the specification.

Relationship to Docker

Nucleus is not a drop-in Docker replacement. It intentionally omits the OCI image spec, registries, and layered union mounts. While Docker focuses on image distribution and portability, Nucleus focuses on hardened sandboxing, policy enforcement, and reproducibility on a single host. It is designed for those who need to run untrusted or ephemeral workloads with stronger, auditable isolation than standard containers provide.

Sources