varunvasudeva1/llm-server-docs

End-to-end documentation to set up your own local & fully private LLM server on Debian. Equipped with chat, web search, RAG, model management, MCP servers, image generation, and TTS.

What it solves

This project provides a comprehensive, end-to-end guide for setting up a private, local LLM server on Debian Linux. It addresses the complexity of coordinating multiple AI components—such as inference engines, chat interfaces, and search tools—while ensuring the system remains secure, stable, and maintainable for users who may be beginners to Linux.

How it works

The guide walks the user through a modular software stack that integrates several open-source tools:

  • Inference: Uses Ollama, llama.cpp, or vLLM to run the models.
  • Interface: Employs Open WebUI as the primary chat platform.
  • Capabilities: Integrates SearXNG for web search, Kokoro FastAPI for text-to-speech, and ComfyUI for image generation.
  • Infrastructure: Uses Docker for containerization, Caddy as a reverse proxy, and Tailscale for secure remote access.
  • System Optimization: Includes scripts for GPU power limiting to reduce energy consumption and systemd for service management.

Who it’s for

It is designed for individuals wanting to host their own AI infrastructure locally, specifically those using Debian (or similar Linux distributions) and those who may be new to server administration.

Highlights

  • Full-Stack Integration: Covers everything from OS installation and GPU drivers to high-level AI applications.
  • Privacy-Focused: Emphasizes a fully private setup where data does not leave the local machine.
  • Modular Design: Encourages using standard protocols (like OpenAI-compatibility and MCP) so components can be easily swapped.
  • Hardware Agnostic: Provides instructions for Nvidia GPUs, AMD GPUs, and CPU-only setups.
  • Security-First: Includes steps for firewall configuration, Docker network isolation, and secure remote access via VPN.