ray-project/kuberay

What it solves

KubeRay simplifies the deployment and management of Ray applications on Kubernetes, removing the complexity of manually configuring and scaling distributed AI workloads.

How it works

It operates as a Kubernetes operator that provides three primary custom resource definitions (CRDs) to manage different workload types:

• RayCluster: Manages the full lifecycle of a Ray cluster, including creation, deletion, autoscaling, and fault tolerance.
• RayJob: Automates the cluster creation and job submission process, with the option to automatically delete the cluster after the job completes.
• RayService: Combines a RayCluster with a Ray Serve deployment graph to enable high availability and zero-downtime upgrades.

Additionally, it offers a kubectl ray plugin for simplified workflows, an API server for configuration management, and an experimental dashboard for resource visualization.

Who it’s for

Infrastructure engineers and ML platform teams who need to run distributed machine learning training, LLM online inference, and batch inference at scale on Kubernetes.

Highlights

• Automated Lifecycle Management: Handles everything from cluster setup to teardown for jobs.
• High Availability: Provides zero-downtime upgrades via RayService.
• Kubernetes Integration: Integrates with observability tools (Prometheus, Grafana), queuing systems (Volcano, Kueue), and ingress controllers (Nginx).
• Scalability: Proven to scale to thousands of nodes for LLM training and inference.