ome-projects/ome

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

What it solves

OME (Open Model Engine) simplifies the complex process of deploying and managing Large Language Models (LLMs) at an enterprise scale. It removes the manual effort required to match models with the right hardware and software runtimes, optimizes GPU usage to reduce costs, and automates the lifecycle of model serving on Kubernetes.

How it works

OME operates as a Kubernetes operator that uses custom resources to define models, runtimes, and hardware. The system automatically parses model files to determine their architecture and size, then uses a weighted scoring system to select the best runtime (such as SGLang, vLLM, or Triton). It integrates with the Kubernetes ecosystem (including Kueue and KEDA) to handle advanced scheduling, autoscaling, and traffic routing, while using AcceleratorClass resources to match workloads to the most cost-effective or capable GPUs.

Who it’s for

It is designed for platform engineers and ML engineers who need to deploy production-grade LLM inference services across Kubernetes clusters and want to automate resource optimization and runtime selection.

Highlights

  • Intelligent Runtime Selection: Automatically matches models to the best inference engine based on quantization, format, and architecture.
  • Enterprise Model Management: Supports distributed storage, double encryption, and automated repair for model files across 80+ model families.
  • Advanced Deployment Patterns: Enables prefill-decode disaggregation and multi-node inference for high-performance serving.
  • GPU Bin-Packing: Uses specialized scheduling to maximize cluster efficiency and ensure high availability.
  • Automated Benchmarking: Includes a built-in BenchmarkJob resource to test performance under various traffic patterns.

Related

  • Project
    kaito-project/kaito

    A Kubernetes operator suite that automates the deployment, scaling, and management of LLM inference, fine-tuning, and RAG engines.

  • Project
    kubeai-project/kubeai

    A Kubernetes operator and proxy that simplifies deploying and scaling AI models with prefix-aware load balancing to optimize LLM performance.

  • Project
    llm-d/llm-d

    llm‑d is an open‑source, CNCF‑sandbox project that adds production‑grade routing, KV‑cache tiering, and expert‑parallelism to existing LLM model servers (vLLM, SGLang). Delivered as Helm charts for Kubernetes, it lets organizations serve large language models at scale with lower latency and higher throughput across many accelerator types.

  • Project
    scaleapi/llm-engine

    An open-source engine for fine-tuning and serving large language models, providing tools to deploy models on hosted or self-managed Kubernetes infrastructure.

  • Project
    volcano-sh/kthena

    A lightweight, Kubernetes-native LLM inference platform that provides modular control and data planes for scalable, cost-efficient model serving.