ome-projects/ome
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
What it solves
OME simplifies the complex process of deploying and managing Large Language Models (LLMs) at an enterprise scale. It removes the manual effort required to match models with the right hardware, select the best inference engine, and manage GPU resources efficiently across a Kubernetes cluster.
How it works
OME operates as a Kubernetes operator that treats models as first-class resources. It automatically parses model files to understand their architecture and size, then uses a weighted scoring system to match the model to the optimal runtime (such as SGLang, vLLM, or Triton) and the best available GPU hardware (via AcceleratorClass). It manages the entire lifecycle from downloading models to deploying them using advanced patterns like prefill-decode disaggregation and multi-node inference, while integrating with the Kubernetes ecosystem for scaling and traffic routing.
Who it’s for
Platform engineers and ML Ops teams who need to deploy production-grade LLM services on Kubernetes and want to automate resource optimization and runtime selection.
Highlights
- Intelligent Runtime Selection: Automatically matches models to the best inference engine based on architecture and quantization.
- Resource Optimization: Uses specialized GPU bin-packing and hardware-aware scheduling to maximize cluster efficiency.
- Advanced Deployment Patterns: Supports multi-node inference and prefill-decode disaggregated serving.
- Integrated Benchmarking: Includes a built-in BenchmarkJob resource to test model performance under various traffic patterns.
- Broad Model Support: Pre-configured for over 80 model families, including Llama, Qwen, and DeepSeek.