Kinference: Open Source Inference Control Plane for vLLM and Kubernetes

Buoyant has released Kinference (pronounced "kay-inference"), an open source control plane for running vLLM inference platforms on Kubernetes. Kinference sits in front of vLLM workers and adds the platform-level features that plain Kubernetes lacks: KV-cache-aware load balancing, a YAML policy engine for routing and cost control, and per-tenant inference metrics. In Buoyant's public benchmarks, KV-cache-aware routing cut median time-to-first-token by 95% and median end-to-end latency by 45% compared with standard Kubernetes load balancing on one workload type.

Kinference is fully open source under Apache v2 and is available on GitHub at Kinference-AI/kinference. It is designed as the control plane layer of a platform you build yourself, with vLLM performing the underlying inference.

What Kinference is and where it fits

Kinference is a control plane, not an inference engine. It uses vLLM for the actual model serving and provides the surrounding platform features needed to operate open weight models in practice: token rate limiting, tenant usage and cost tracking, intelligent load balancing, and inference policies.

Adoption requires no change to inference clients beyond pointing them at Kinference's HTTP endpoint. Kinference accepts OpenAI- or Anthropic-compatible inference traffic, so existing client libraries and applications can be redirected to it without modification.

The project was created at Buoyant after observing customers struggle to run open weight models in their own environments. Buoyant frames the core problem as serving "a heterogeneous, evolving set of models running on a heterogeneous, evolving pool of GPUs, to a heterogenous, evolving set of clients."

Kinference ships as two independently released components: a traffic gateway and an inference handler. Each ships its CRDs as a separate Helm chart that must be installed before the component itself.

KV-cache-aware load balancing

Kinference's load balancer routes on inference-specific signals rather than generic connection-level ones. It takes into account GPU queue sizes, historical performance, and the contents of the KV cache on each GPU.

KV-cache awareness is the key differentiator: by modeling what is already cached on a given GPU, Kinference can send a request to the vLLM worker that will do the least additional work for that specific request. This matters most for workloads with shared prefixes and multi-turn conversations, where a large fraction of the prompt may already be resident in a worker's cache.

Benchmark results

Buoyant published two benchmark comparisons against standard Kubernetes load balancing (labeled "service" in the charts, versus "kinference").

Unloaded latency benchmark — designed to isolate the effect of KV-cache-aware load balancing:

  • Median time-to-first-token (TTFT) reduced by 95%
  • Median end-to-end latency reduced by 45%

These figures apply to one workload type in that benchmark.

Capacity benchmark — measured requests per minute each endpoint can serve within a latency SLO:

  • 7.4× the goodput on the shared-prefix workload
  • About 3.5× the goodput on the multiturn workload
  • Kinference attained the chosen SLO at concurrency 32 (the highest measured) on shared-prefix and at concurrency 8 on multiturn
  • The kube-proxy approach did not attain the SLO at any concurrency level

Full details on benchmarking setup, process, and findings are published in Buoyant's separate benchmark write-up.

Inference policies: a policy-driven routing layer

Kinference's central abstraction is the inference policy. Rather than shipping a set of one-off features, Buoyant built a policy layer intended to express many routing, cost, and priority requirements in a single mechanism.

Policies let platform owners balance tradeoffs across usage, performance, and priorities. Buoyant cites two recurring customer questions that motivated the design:

  • How can users request broader classes of models — "any Qwen 3.8 model" or "whatever is good for coding" — rather than a specific checkpoint such as "Qwen 3.8 with 27B params and the FP8 quantization"?
  • How can "easy" requests be redirected to cheaper models even when the user requested an expensive one, without sending the whole prompt history to a cold GPU and incurring a large latency hit?

Policies are declared as Kubernetes custom resources (kind: InferencePolicy, apiVersion: kinference.buoyant.io/v1alpha1) and can route or reject requests based on tenant, requested model, the state of the underlying GPU pool, semantic classification of prompts, and other factors. Because they are declarative YAML, policies can go through human or agentic review, be committed to git, be linted and vetted, and be applied or rolled back independently.

Example: semantic routing to cheaper models

The following policy semantically classifies prompts by complexity and routes simpler prompts to cheaper models — unless, accounting for cache usage, it is faster to send the request to the expensive model:

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata:
  name: smart-semantic-routing
  namespace: inference
spec:
  rules:
  - name: low-complexity-consider-qwen
    matches:
    - model: glm-5.3
      tag: complexity-low
    route:
      consider:
      - models: { names: [ qwen-3.6-27b, glm-5.3 ] }
      select: [ { minimize: PredictedTTFT } ]

Example: bursting to cloud providers

This policy bursts excess requests to third-party provider endpoints (Buoyant names OpenAI, Anthropic, and Databricks as examples) when queue depth or cache pressure indicates the GPUs are saturated:

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata:
  name: burst-policy
  namespace: inference
spec:
  rules:
  - name: burst-to-cloud
    route:
      exclude:
      - name: drop-overloaded-workers
        drop:
          backend:       { type: Local }
          queuedPrompts: { greaterThan: 2 }
 
      select:
      - { type: Local,
          minimize: PredictedTTFT }
      - { type: Provider,
          order: [bedrock, databricks] }

Example: keeping PII inside the cluster

This policy classifies prompts by whether they contain PII and prevents such requests from ever leaving the cluster, failing closed if no local backend is available:

apiVersion: kinference.buoyant.io/v1alpha1
kind: InferencePolicy
metadata:
  name: pii-stays-local
  namespace: inference
spec:
  rules:
  - name: serve
    route:
      exclude:
      - name: pii-never-leaves-cluster
        if:   { tag: pii-detected }
        drop: { backend: { type: Provider } }
      select:
      - { type: Provider,
          order: [ databricks ] }
      - { type: Local,
          minimize: PredictedTTFT }
      onEmpty: Reject   # fail closed

Inference metrics and SLOs

Kinference exposes a suite of inference-specific metrics for monitoring, alerting, and SLO construction. Reported metrics include:

  • Token rates in and out, plus histograms of input and output token sizes
  • TTFT, ITL (inter-token latency), and end-to-end latency histograms
  • Uncached versus cached prompt tokens, for cost estimation
  • Model usage and GPU saturation

Almost every metric is also available per tenant, which allows operators to track tenant usage, build tenant-specific performance SLOs, and verify that the platform meets its requirements for each consumer.

Installing Kinference

The published quickstart installs Kinference into the default namespace; other namespaces require overriding the namespaces in the Helm commands. The steps below reflect the versions and chart locations given in the announcement, and Buoyant notes that the official documentation at kinference.ai/docs should be treated as the current source of truth as the steps evolve.

1. Install the Gateway API CRDs

GWAPI_VERSION=1.5.1
kubectl apply --server-side --force-conflicts -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v$GWAPI_VERSION/standard-install.yaml

2. Install the Kinference gateway

helm upgrade -i gateway-crds \
  oci://ghcr.io/buoyantio/charts/gateway-crds \
  --devel
 
helm upgrade -i gateway \
  oci://ghcr.io/buoyantio/charts/gateway \
  --devel --wait

3. Install Kinference

helm upgrade -i kinference-crds \
  oci://cr.kinference.ai/kinference-ai/charts/kinference-crds \
  --devel
 
helm upgrade -i kinference \
  oci://cr.kinference.ai/kinference-ai/charts/kinference \
  --set extension.enabled=true \
  --devel --wait

4. Verify the installation

Run kubectl get pods and confirm both processes are running:

NAME                           READY   STATUS    RESTARTS   AGE
gateway-cp-566cddc46d-jsqlq    1/1     Running   0          8s
kinference-7bb854cfc5-pp4t7    1/1     Running   0          8s

Why it matters

Kinference addresses a gap that appears once open weight models move from a single vLLM deployment into a shared, multi-tenant platform. Kubernetes' default load balancing is connection-oriented and blind to inference state, which the capacity benchmark illustrates directly: the kube-proxy approach failed to meet the latency SLO at any tested concurrency level, while Kinference met it at the highest measured concurrency on the shared-prefix workload.

By making routing decisions declarative YAML rather than hard-coded features, Kinference also makes inference platform behavior reviewable and reversible through standard Git and Kubernetes workflows — including safety-relevant rules such as keeping PII inside the cluster with a fail-closed default.

Buoyant explicitly invites feedback, issues, and benchmark reports from other deployments via the project's GitHub repository.

Sources