kube32: Kubernetes-compatible control plane for ESP32 microcontrollers

Overview

kube32 is an open-source project that implements a Kubernetes-compatible control plane and WebAssembly (Wasm) runtime specifically for ESP32-S3 microcontrollers. It allows developers to manage a cluster of embedded devices using standard tools like kubectl and k9s, enabling cloud-native orchestration patterns for distributed edge computing.

Architecture and Core Functionality

The system uses the wasm3 interpreter to execute pods as WebAssembly modules. A single firmware image is deployed across all nodes, with roles determined by hardware capabilities:

  • Control Plane: Runs on ESP32-S3 boards equipped with at least 2 MB of PSRAM. It manages the cluster state, serves the Kubernetes API, and handles scheduling.
  • Worker Nodes: Run on standard ESP32-S3 boards. They execute pods and report status back to the control plane.
  • Simulator: The kube32-sim tool allows developers to build and test multi-node clusters on macOS or Linux before deploying to physical hardware.

Kubernetes Compatibility

kube32 provides extensive support for standard Kubernetes primitives, enabling a familiar workflow for those experienced with cloud-native development. Supported features include:

  • API Resources: Pods, Nodes, Namespaces, Services, Deployments, DaemonSets, Jobs, ConfigMaps, Secrets, and Leases.
  • Operational Tooling: Full support for kubectl and k9s, including watch streams, table output, OpenAPI v3 validation, and log tailing.
  • Networking: Service routing, HTTP Ingress with host/prefix rules, and synchronous pod-to-pod HTTP calls.
  • Lifecycle Management: Support for restartPolicy (Always, OnFailure, Never), exponential backoff for crashes, and OOM (Out of Memory) tracking.

WebAssembly Pods and Security

By leveraging WebAssembly, kube32 provides a sandboxed environment for embedded applications. Developers can write pods in Rust, C, or AssemblyScript. The runtime enforces strict resource management, including configurable memory limits and per-call gas budgets to prevent individual pods from destabilizing the node.

Security is further reinforced through a capability firewall that gates access to hardware peripherals, such as GPIO pins, I2C/UART buses, and microSD storage, on a per-pod basis. Communication between nodes is handled via ESP-NOW and UDP, with optional TLS support for API traffic.

Community Discussion and Perspectives

The release of kube32 has prompted discussion regarding the intersection of cloud-native paradigms and embedded systems. While the project demonstrates the technical feasibility of running Kubernetes-like orchestration on constrained hardware, some community members have questioned the practical application of this architecture.

In response to skepticism about the utility of "cloud-native embedded devices," the project author emphasized the potential for distributed edge computing:

"I had a few esp32 boards around for a while and always thought - how do we go beyond just one device, would it be possible to apply cloud-native thinking to distributed edge computing? It opens so many possibilities for distributed edge computing and multi-device systems."

Getting Started

Users can begin by building the simulator or flashing hardware via PlatformIO. The project includes a reference application called "Tessera," an issue tracker that distributes pods across multiple nodes to demonstrate service routing, persistent storage on microSD cards, and real-time synchronization. Full documentation, including hardware specifications and API coverage, is available at the official project documentation site.

Sources