mudler/LocalAI

LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.

LocalAI – Open‑source, on‑premise AI engine

What it is – LocalAI is a modular, self‑hosted server that lets you run a huge variety of AI models (large‑language‑models, vision, speech‑to‑text, text‑to‑speech, image‑generation, video, etc.) on any hardware – from a laptop CPU to a multi‑GPU server, Apple Silicon, AMD/Intel GPUs or even Jetson boards. The core is tiny; each model type is served by a backend that is pulled on‑demand as a separate OCI image (llama.cpp, vLLM, whisper.cpp, stable‑diffusion, …). This keeps the install lightweight and lets you add only the pieces you need.

Key capabilities (as described in the README)

Capability	Details
Model support	LLMs, vision, audio, image, video. Load models from the LocalAI gallery, Hugging Face, Ollama OCI, or any OCI registry.
Hardware flexibility	Runs on CPU‑only or with GPU acceleration: NVIDIA CUDA 12/13, AMD ROCm, Intel oneAPI, Apple Metal, Vulkan, Jetson L4T.
Composable backends	Each backend is a separate container image (llama.cpp, vllm.cpp, parakeet.cpp, etc.) fetched only when required.
API compatibility	Exposes OpenAI, Anthropic and ElevenLabs‑compatible REST endpoints, plus a Realtime API for speech‑to‑speech.
Multi‑user & security	API‑key authentication, role‑based quotas, per‑user usage metrics, optional OIDC, deny‑by‑default auth.
Built‑in agents	Autonomous agents with tool use, Retrieval‑Augmented Generation (RAG), Model Context Protocol (MCP) and a community‑driven Agent Hub.
Privacy‑first	All inference runs on‑prem, data never leaves your infrastructure.
Extensible	Open interface for writing new backends in any language; many native C/C++ GGML backends are provided (vllm.cpp, parakeet.cpp, voice‑detect.cpp, etc.).
Distributed mode	Horizontal scaling with PostgreSQL + NATS, smart routing, replica management, and resumable uploads.
Web UI & CLI	Integrated web UI, terminal‑based chat agent, and `local‑ai` CLI for model management, benchmarking, and interactive chatting.

Typical workflow (quick‑start)

Run the server – e.g. docker run -ti -p 8080:8080 localai/localai:latest (or use the macOS DMG). The container automatically detects your GPU and pulls the appropriate backend.
Load a model – local‑ai run llama-3.2-1b-instruct:q4_k_m or point to a Hugging Face URL.
Talk to it – local‑ai chat --model llama-3.2-1b-instruct:q4_k_m or call the OpenAI‑compatible HTTP API from any client.
Add capabilities – enable vision, TTS, or an autonomous agent simply by pulling the corresponding backend image.

Who might use it?

Developers & startups that need an on‑premise LLM or multimodal service without vendor lock‑in.
Enterprises with strict data‑privacy policies.
Researchers wanting to experiment with many backends (llama.cpp, vLLM, MLX, etc.) from a single interface.
Edge deployments on Jetson, Apple Silicon, or low‑power CPUs.

Recent highlights (2026 releases)

v4.11.0 – Audio scene analysis, Decisions API, model failover chains, signed OCI galleries, text‑to‑animation.
v4.10.0 – Fleet dashboard, unified benchmark tool, private model credentials.
v4.9.0 – Deny‑by‑default auth, context compression, MiniMax‑H3 video‑plus‑audio generation.
v4.8.0 – vllm.cpp (alpha), image‑to‑3D, new audio.cpp backend for speech and sound‑event detection.
v4.7‑4.5 – Voice cloning, video generation, tool‑calling agents, biometric backends (voice‑detect, face‑detect), realtime streaming APIs.

Bottom line

LocalAI is a genuine, actively maintained open‑source platform that brings together a wide range of AI models and hardware accelerators behind a single, OpenAI‑compatible API. Its modular, on‑demand backend architecture makes it lightweight yet extensible, and the project includes many production‑grade features such as multi‑user auth, distributed scaling, and built‑in autonomous agents.