open-webui/oikb

What it solves

oikb is a synchronization tool designed to keep Open WebUI Knowledge Bases updated automatically. It eliminates the manual effort of uploading files to a knowledge base by connecting various external data sources to Open WebUI, ensuring the LLM has access to the most current information.

How it works

The tool uses incremental SHA-256 diffing to ensure only new or modified files are uploaded. The process follows four steps:

1. Scanning the source for files and computing checksums.
2. Sending a manifest to the Open WebUI /sync/diff endpoint.
3. Deleting stale files and creating missing directories.
4. Uploading only the files that have changed.

Who it’s for

Users of Open WebUI (version 0.9.6+) who manage large amounts of documentation, code, or research data across multiple platforms (like GitHub, Confluence, or S3) and want to integrate this data into their AI knowledge bases without manual updates.

Highlights

• Extensive Connectivity: Supports 46 connectors across categories like code repos, cloud storage, wikis, ticketing systems, messaging apps, and research tools (e.g., Zotero).
• Hedge-case Management: Includes filters for include/exclude globs and file size limits to control exactly what gets synced.
• Production-Ready Daemon: Offers a long-lived scheduler with an HTTP API, Prometheus metrics, health checks, and failure notifications via Slack or PagerDuty.
• LLM-Triggered Syncs: Can be added as an OpenAPI tool server in Open WebUI, allowing the LLM itself to trigger syncs and check status.
• Flexible Deployment: Supports CLI usage, Docker, Docker Compose, and GitHub Actions.