ome-projects/ome

OME (Open Model Engine) – Kubernetes Operator for LLM Serving

What it is – OME is a Kubernetes‑native controller that lets you run large language models (LLMs) at scale inside a cluster. It treats each model, runtime, and GPU resource as a first‑class Kubernetes custom resource, automating the whole lifecycle from model ingestion to inference‑service rollout.

Key capabilities

• Model CRDs – BaseModel/ClusterBaseModel describe a model file (architecture, parameter count, format). OME parses the file, supports many formats (SafeTensors, PyTorch, TensorRT, ONNX) and stores models with double encryption and namespace scoping.
• Intelligent runtime selection – OME scores available runtimes (e.g., SGLang, vLLM) against a model’s characteristics and picks the best one automatically.
• Optimized deployment patterns – Supports pre‑fill/‑decode disaggregation, multi‑node inference, canary/blue‑green rollouts, and custom autoscaling policies.
• GPU‑aware scheduling – AcceleratorClass resources describe GPU capabilities and cost; OME’s bin‑packing scheduler packs workloads to maximise cluster utilisation while respecting HA requirements.
• Fleet‑level quota (alpha) – Optional control‑plane components (ome-quota-manager, ome-alfred) build a hierarchical GPU‑quota tree and surface it to Kueue for gang‑scheduling across large GPU farms.
• CLI & web console – kubectl ome (a kubectl plugin) lets operators inspect models, runtimes, rollouts, and issue guarded actions (pause, promote, rollback, migration). A separate web UI (ome-console) provides a visual dashboard with HuggingFace model search.
• Benchmarking – BenchmarkJob CRD runs configurable traffic loads and stores results, making performance comparison repeatable.

How to get it

1. Prerequisite – Kubernetes 1.28+.
2. Install CRDs & core operator – via Helm from an OCI registry:

   helm upgrade --install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd --namespace ome --create-namespace
   helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources --namespace ome
   

3. Optional add‑ons – scheduler, quota manager, and the “Alfred” caretaker are each separate Helm charts you can enable as needed.
4. CLI – install the kubectl ome plugin with Krew:

   kubectl krew install --manifest-url https://github.com/ome-projects/ome/releases/latest/download/ome.yaml
   

   Or build from source (make kubectl-ome).

Typical workflow

1. Create a BaseModel (or ClusterBaseModel) pointing at a model file stored in a supported backend.
2. OME parses the model, creates a ServingRuntime (or picks an existing one) and matches it to an AcceleratorClass.
3. An InferenceService resource ties the model to the runtime; OME generates the necessary Pods, Services, and Gateway‑API routes.
4. Autoscaling policies (via KEDA or HPA) and rollout policies keep the service responsive and allow safe updates.
5. Use kubectl ome or the web console to monitor health, view placement decisions, and run BenchmarkJobs.

Production readiness – The project ships a v1beta1 API, full documentation, unit/integration test coverage, RBAC‑based security, model encryption, and HA storage. It is already used in production for large‑scale LLM workloads.

License – Apache 2.0.

---

TL;DR – OME is a production‑grade, Kubernetes‑native platform that automates everything needed to serve LLMs—model ingestion, runtime selection, GPU‑aware scheduling, rollout strategies, and observability—through declarative CRDs, a Helm‑installable operator, a CLI, and a companion web UI.