InftyAI/llmaz

What it solves

llmaz is a production-ready inference platform designed to simplify the deployment and management of large language models (LLMs) on Kubernetes. It removes the complexity of manually configuring infrastructure, model loading, and scaling for high-performance AI services.

How it works

llmaz acts as an orchestration layer on Kubernetes that integrates with various state-of-the-art inference backends (such as vLLM, TensorRT-LLM, and llama.cpp). It automates the model loading process from providers like HuggingFace and ModelScope and manages the lifecycle of the model service. It also supports distributed inference and integrates with the AI Gateway for traffic management and scaling tools like Karpenter for node autoscaling.

Who it’s for

Cloud engineers, MLOps engineers, and developers who need to deploy LLMs into a production environment on Kubernetes with minimal configuration and overall infrastructure overhead.

Highlights

• Broad Backend Support: Compatible with vLLM, Text-Generation-Inference, SGLang, llama.cpp, and TensorRT-LLM.
• Automated Model Loading: Handles model weights from HuggingFace, ModelScope, and ObjectStores automatically.
• Heterogeneous Cluster Support: Can serve models across different types of devices using the InftyAI Scheduler.
• Scaling Efficiency: Supports horizontal pod scaling based on LLM metrics and node autoscaling via Karpenter.
• Integrated AI Gateway: Provides token-based rate limiting and model routing through Envoy AI Gateway.
• Built-in ChatUI: Integrates with Open WebUI for out-of-the-box chatbot capabilities including RAG and function calling.