Modal: Solving the 100,000 Sandbox Problem for AI Agents

The Shift from Developer Experience to Agent Experience

Modal is transitioning its focus from Developer Experience (DX) to Agent Experience (AX). The core thesis is that the benefits of high-quality DX—such as reducing boilerplate and simplifying infrastructure provisioning—apply directly to AI agents. Instead of requiring an agent to navigate hundreds of untyped Kubernetes YAML files, Modal allows agents to make changes via decorators in code, enabling a self-provisioning runtime where agents can see changes live in action.

As agents increasingly write the code, the role of the human shifts toward observability. While agents can use the CLI for investigation, humans are still required to interpret dashboards and make high-level judgment calls. Consequently, high-quality observability tools are now more critical than the ability to read the underlying code.

Why Traditional Cloud Infrastructure Fails AI Workloads

Kubernetes and traditional cloud providers were designed for steady-state web server use cases and slow scaling. AI workloads, however, are characterized by extreme "burstiness" and a need for specialized compute (accelerators) and custom images.

Modal addresses these gaps by providing:

• Elastic Inference: The ability to scale rapidly from zero to thousands of GPUs across different regions to match unpredictable traffic patterns.
• Serverless Primitives: A runtime that allows compute-heavy workloads to burst up and down without the management overhead of traditional clusters.
• Multi-Cloud Capacity: A "supercloud" strategy spanning 17 different cloud providers, allowing Modal to optimize for data locality, latency, and reliability by building a software-defined reliability layer on top of various NeoClouds.

Solving the 100,000 Sandbox Problem

For specific AI workloads, particularly Reinforcement Learning (RL) rollouts, the infrastructure requirements are massive and highly volatile. Akshat Bubna notes that RL rollouts can require up to 100,000 sandboxes simultaneously.

To support this, Modal has developed several key technical primitives:

• GPU Snapshotting: By taking snapshots of the GPU state (e.g., torch compiler model snapshots), Modal significantly reduces cold start times, allowing for faster scaling during bursty inference or training runs.
• Networked Sandboxes and Sidecars: Modal supports "sidecars," allowing a sandbox to be a pod of multiple containers. This is essential for running man-in-the-middle proxies for logging or controlling egress networking.
• Private IPv6 Overlay (I6PN): An overlay network that allows containers within the same workspace to address each other using private IPv6 addresses, which is critical for distributed training.
• RDMA Networking: To bypass the TCP networking stack and transfer memory faster between nodes, Modal provides RDMA networking (up to 3 terabits per second), which is essential for multi-node post-training of medium-sized models.

Advancing Inference Performance with DeFlash

Modal is moving beyond being a "black box" for deployment to optimizing the model layer itself. A primary focus is increasing inference speed through speculative decoding.

Speculative Decoding and DeFlash

Speculative decoding uses a smaller "draft model" to predict tokens ahead of a larger model, which then verifies them in batch. This approach is more efficient because it leverages compute rather than being bound by memory bandwidth.

Modal has open-sourced DeFlash, a block-based speculator. While kernel optimizations only provide marginal percentage gains, increasing the "accept length" (the number of tokens the big model accepts from the draft model) can result in a multiplicative speedup of 2x to 4x without sacrificing quality.

Auto Endpoints

To make frontier-level performance accessible without requiring deep expertise, Modal introduced Auto Endpoints. These allow users to create optimized endpoints via the UI or CLI with DeFlash and other optimizations baked in, while remaining transparent; users can eject into the full Modal experience to tweak models or fine-tune configurations.

The Future of AI Infrastructure: Auto-Research and CI

Modal is expanding into "auto-research," where agents are given GPUs to perform hyperparameter sweeps guided by model intuition. This is more efficient than traditional grid searches and allows agents to autonomously tweak configurations, run profilers, and switch GPU types (e.g., from H200 to B200) to find optimal performance.

Additionally, Modal sees a significant opportunity in the CI (Continuous Integration) market. As coding agents increase the volume of CI runs, the current process of preparing dependencies and artifacts is wasteful. Modal believes primitives like memory snapshotting and restoration can make CI significantly more efficient for agentic workflows.