kserve/kserve

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

What it solves

KServe provides a standardized way to deploy and scale both generative and predictive AI models on Kubernetes. It eliminates the complexity of managing multi-framework deployments and handles the infrastructure needs of enterprise-scale AI workloads, such as autoscaling and resource optimization.

How it works

KServe acts as an inference platform that unifies different AI workloads. For generative AI, it uses optimized backends like vLLM and llm-d and supports OpenAI-compatible protocols. For predictive AI, it supports multiple frameworks (TensorFlow, PyTorch, scikit-learn, etc.) and uses an InferenceGraph to manage request routing between predictors, transformers, and explainers. It can be installed as a lightweight standalone service, a serverless deployment via Knative, or a high-density deployment via ModelMesh.

Who it’s for

It is designed for organizations and developers who need to deploy AI models at scale on Kubernetes, ranging from those needing quick deployments to enterprises requiring advanced traffic management and cost-efficient resource scaling.

Highlights

  • Unified Inference: Supports both Generative AI (LLMs) and Predictive AI (traditional ML) on one platform.
  • Generative AI Optimizations: Includes GPU acceleration, KV cache offloading to CPU/disk, and intelligent model caching.
  • Predictive AI Tooling: Offers built-in model explainability, canary rollouts, and advanced monitoring for drift and outlier detection.
  • Efficient Scaling: Supports request-based autoscaling and "scale-to-zero" to reduce infrastructure costs when models are not in use.
  • Broad Compatibility: Native support for Hugging Face models and various ML frameworks like PyTorch, TensorFlow, and ONNX.

Related

  • Project
    llm-d/llm-d

    A high-performance distributed inference serving stack for Kubernetes that optimizes LLM production deployments across various hardware accelerators.

  • Project
    volcano-sh/kthena

    A Kubernetes-native LLM inference platform that provides declarative model lifecycle management and intelligent routing to make AI infrastructure scalable and cost-efficient.

  • Project
    SeldonIO/MLServer

    An open-source inference server that provides a standardized REST and gRPC interface for serving machine learning models across various frameworks.

  • Project
    ai-dynamo/dynamo

    An open-source, datacenter-scale orchestration layer that coordinates multiple inference engines across GPU clusters to maximize throughput and minimize latency.

  • Project
    kubeflow/trainer

    A Kubernetes-native distributed AI platform for scalable LLM fine-tuning and training across various frameworks like PyTorch, JAX, and XGBoost.