intentee/paddler
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
What it solves
Paddler provides a way to run, deploy, and scale LLMs on your own infrastructure. It addresses the need for privacy, reliability, and cost control by removing dependence on closed-source model providers and avoiding per-token pricing.
How it works
It operates as a self-contained binary with two primary components: a balancer that handles incoming requests, management, and a web admin panel, and agents that are deployed on separate instances to generate tokens and embeddings using a built-in llama.cpp engine. Requests are distributed to slots within these agents, where each slot maintains its own context and KV cache.
Who it’s for
- Product teams integrating LLM inference and embeddings into their features.
- DevOps and LLMOps teams managing LLMs at scale.
- Organizations with high compliance and privacy requirements (e.g., medical or financial sectors).
- Product leaders seeking consistent, predictable model performance.
Highlights
- LLM-specific load balancing: Distributes requests across a fleet of agents.
- Dynamic scaling: Agents can be added dynamically, allowing for integration with autoscaling tools and scaling from zero hosts via request buffering.
- Dynamic model swapping: Ability to change models without downtime.
- Built-in management: Includes a web admin panel for monitoring, testing, and configuring models and chat templates.
- Flexible deployment: Available as a CLI for infrastructure use or a desktop application for local AI clusters.