zylon-ai/private-gpt

Complete API layer for private AI applications on local models: RAG, skills, tools, MCP, text-to-sql, and more. Works with any OpenAI-compatible inference server.

What it solves

PrivateGPT is an open-source API layer that allows developers to build production AI applications using local models without relying on cloud APIs. It bridges the gap between raw local inference servers (like Ollama or vLLM) and a fully functional AI application by providing high-level building blocks such as RAG, tool use, and structured data access.

How it works

PrivateGPT acts as an intermediary layer between your application or agent and an OpenAI-compatible inference server. It does not run the models itself; instead, it connects to a server via OPENAI_API_BASE to handle the actual LLM processing. It follows the Claude API specification to provide a standardized interface for messages, streaming, and tool orchestration.

Who it’s for

Developers building private, self-hosted AI products, agents, or workflows who want a standardized API for local LLMs and want to avoid rebuilding backend primitives from scratch.

Highlights

  • Claude API Compatibility: Implements a reference API for messages, streaming, and token counting.
  • Agentic RAG: Provides file and artifact ingestion with retrieval and citations.
  • Built-in Tools: Includes web search, web fetch, and code execution, with support for custom tools and MCP connectors.
  • Data Access: Built-in structured access to databases and CSVs for tabular analysis.
  • Workbench UI: Includes a polished UI for testing, document uploading, and API debugging.

Related

  • Project
    arc53/DocsGPT

    An open-source AI platform for building private intelligent agents and assistants that perform enterprise search and document analysis with source citations.

  • Project
    PromtEngineer/localGPT

    A private, on-premise document intelligence platform that uses hybrid search and local LLMs to let users query and summarize their files without data leaving their machine.

  • Project
    schmitech/orbit

    An open retrieval-based inference toolkit that connects LLMs to private data, APIs, and MCP tools through a single, governed OpenAI-compatible endpoint.

  • Project
    orailnoor/cross-platform-llm-client

    A cross-platform AI chat client that unifies local on-device LLM inference via llama.cpp with cloud API access for OpenAI, Anthropic, and Google Gemini.

  • Project
    open-webui/open-webui

    A feature-rich, self-hosted AI platform that provides a user-friendly interface for local and API-based LLMs, featuring built-in RAG and extensive plugin support.