PromtEngineer/localGPT

What it solves

LocalGPT is a private, on-premise document intelligence platform that allows users to query, summarize, and extract insights from their own files without any data leaving their local machine. It addresses the privacy concerns associated with cloud-based AI by providing a fully local RAG (Retrieval-Augmented Generation) system.

How it works

LocalGPT uses a modular architecture combining several advanced retrieval techniques to ensure high-quality answers:

• Hybrid Search: Fuses dense vector search with full-text search using Reciprocal Rank Fusion (RRF).
• Smart Routing: A router determines whether a query requires RAG (retrieving documents) or can be answered directly by the LLM.
• Query Decomposition: Complex questions are split into sub-questions, which are retrieved and synthesized into a final answer.
• Reranking: A cross-encoder reranker refines the candidate set to surface the most relevant content.
• Document Processing: Uses Docling for parsing PDF, DOCX, HTML, Markdown, and TXT files, with OCR fallback for text-less PDFs.
• Local Model Execution: Integrates with Ollama for generation and utility models, and HuggingFace for embedding and reranking models.

Who it’s for

It is designed for individuals or organizations that need to interact with their private documents using AI while maintaining 100% data security and privacy.

Highlights

• Utmost Privacy: 100% on-premise execution; no data ever leaves the machine.
• Advanced Retrieval: Features hybrid search, calibrated reranking, and optional late chunking and sentence pruning.
• Versatile Model Support: Easily swap generation models via Ollama and embedding models from HuggingFace.
• Developer Friendly: Provides RESTful APIs for both the backend and the RAG core, allowing for custom application building.
• Modern Interface: Includes a responsive web UI for index and session management with live streaming of the retrieval process.