ServerlessLLM/ServerlessLLM

What it solves

ServerlessLLM addresses the high cost and latency associated with deploying multiple large language models (LLMs) on shared GPU resources. It eliminates the slow model loading times that typically hinder "serverless" AI deployments, allowing multiple models to be swapped in and out of GPU memory rapidly.

How it works

The system employs three primary technical innovations:

1. Fast Checkpoint Loading: It uses a custom binary storage format and O_DIRECT I/O to bypass the OS page cache, combined with a pinned memory pool for DMA-accelerated transfers to the GPU. This allows models to load 6-10x faster than standard SafeTensors.
2. GPU Multiplexing: It implements storage-aware scheduling and fast switching to allow a single GPU to serve many different models, including the ability to scale instances to zero when idle.
3. Unified Pipeline: It integrates inference with LoRA fine-tuning, enabling the efficient serving of a single base model paired with hundreds of specialized LoRA adapters.

Who it’s for

It is designed for developers and infrastructure engineers who need to host multiple AI models on limited GPU hardware, those building serverless AI platforms, or teams requiring on-demand fine-tuning and inference for many different users/tasks.

Highlights

• Extreme Loading Speed: Loads models 6-10x faster than SafeTensors.
• High Density: Capable of serving 10+ models on a single GPU.
• Broad Compatibility: Works with vLLM, Transformers, and supports both NVIDIA and AMD GPUs.
• Production Ready: Provides an OpenAI-compatible API and supports Docker, Kubernetes, and multi-node clusters.
• RAG Support: Can deploy embedding models alongside LLMs via a dedicated embeddings endpoint.