Knowhere – Turning Unstructured Docs into Agent‑Ready Memory

What it is

Knowhere is an open‑source backend that ingests all kinds of unstructured files (PDF, PowerPoint, Word, images, markdown, etc.), parses them, and stores the results in a hierarchy‑native memory graph that AI agents can query.

• Dual‑track parsing – a Text track that keeps reliable text structure and a Vision track that uses a vision‑language model to understand complex pages (e.g., scanned PDFs or slide decks). Both tracks output the same chunk‑and‑metadata schema.
• Agent‑neutral retrieval – the system exposes a single corpus API with tools for outline navigation, structural filters, exact/fuzzy search, and full‑page reading. Agents decide how and when to call these tools.
• Cross‑document graph – extracted entities, tables, and images become nodes that link related documents, enabling “knowledge‑graph‑style” retrieval.
• Citation‑ready – every result is tied back to its source document, section, page number, and visual evidence.

Why it matters

Traditional RAG pipelines flatten documents into independent text chunks, losing hierarchy, citations, and visual context. Knowhere keeps the original structure and adds visual understanding, so agents can:

• Navigate a document the way a human would (chapter → section → page).
• Retrieve accurate evidence with page numbers and images.
• Work efficiently on ultra‑long PDFs or slide decks that OCR alone would struggle with.

Core components (as described in the repo)

Component	Role
API (apps/api)	HTTP endpoints for uploading files, launching parsing jobs, and querying the memory graph.
Worker (apps/worker)	Background processors that run the parsing pipelines (text or vision) and build the hierarchy graph.
Dashboard (separate repo)	Web UI for managing collections, viewing parsed hierarchies, and testing retrieval.
Self‑hosted stack (Docker‑Compose)	Quick local deployment of API, worker, Postgres, Redis, and optional S3 storage.
SDKs (Python/Node)	Thin client libraries that wrap the API for easy integration.

How to get started (quick‑start steps)

1. Install dependencies – Python 3.11+, uv, Docker‑Compose.
2. Sync Python packages: uv sync --all-packages.
3. Configure – copy the .env.example files for the API and worker, then fill in:
   • DB/Redis connections
   • S3‑compatible storage credentials
   • At least one LLM provider key (e.g., DeepSeek‑Flash, OpenAI, Qwen, etc.)
   • Vision model credentials for the Vision track.
4. Launch the stack: ./deploy/local-dev/start-dev.sh (starts Postgres, Redis, LocalStack, etc.).
5. Run migrations and start the API and worker in separate terminals.
6. Use the API – upload a document via the V2 Jobs endpoint; the system will route it to the appropriate track, build the hierarchy, and make it searchable.

Typical workflow for an AI agent

1. Ingest – Agent sends a file to the Knowhere API.
2. Parse – Worker runs either the Text or Vision pipeline, normalises output into a common schema.
3. Store – Nodes (sections, pages, images, tables) are persisted with metadata (paths, citations, entities).
4. Retrieve – Agent calls the retrieval API, using tools like outline, search, or read_page to explore the graph.
5. Cite – Results include precise source references, enabling trustworthy answers.

Notable features highlighted in the README

• Vision Page Understanding – no need for perfect OCR; a VLM can summarise a whole page.
• Ultra‑long PDF support – handles hundreds of pages and atlas‑style documents.
• Cross‑document memory graph – typed entities link related docs across a namespace.
• Benchmarks – internal tests show +36 % first‑try accuracy and +11 % recall vs. raw‑document baselines.
• Telemetry – anonymous usage data sent to PostHog (opt‑out via env var).

Who might use it

• Developers building agentic RAG systems that need structured, citation‑ready context.
• Enterprises with large internal document libraries (manuals, slide decks, research PDFs) wanting a searchable knowledge base.
• Researchers who need reproducible document parsing pipelines and want to cite exact pages.

License & community

• Apache 2.0 license.
• Active GitHub Discussions, issue tracker, and contribution guide.
• Docker images are published on GitHub Container Registry.
• The project is open‑source as of May 2026, with a managed cloud offering for quick trials.

---

All details above are taken directly from the repository’s README; no additional claims have been added.