xberg-io/xberg

Xberg – a multi‑language document‑intelligence engine

What it is – Xberg is an open‑source library and CLI that turns any kind of file (PDF, Office docs, images, audio, archives, source code, web pages, etc.) into clean, structured text. It bundles format detection, OCR, layout analysis, table reconstruction, code parsing, and optional LLM‑based enrichment (summarisation, NER, embeddings, schema‑driven JSON extraction). The core is written in Rust and exposed through 15 language bindings (Python, Node, Go, Java, C#, Ruby, PHP, Elixir, Dart, Swift, Kotlin, Zig, C FFI, etc.) and can be run as a library, a command‑line tool, a REST API, or a “MCP” server for AI‑coding assistants.

---

Key capabilities (as described in the README)

Capability	What you get
Broad format support	107 document formats across 141 extensions (PDF, Office, e‑books, images, audio, archives, email, code, scientific files, etc.).
Web ingestion	Fetch a single URL or crawl a site (via the companion crawlberg engine).
Audio/video transcription	Whisper‑ONNX models (tiny → large‑v3) turn MP3, WAV, MP4, WebM, etc. into text.
Recursive archive extraction	Handles nested .zip, .tar, .gz, .7z with safety limits (zip‑bomb protection, depth caps).
OCR back‑ends	Tesseract, PaddleOCR, Candle, or VLM‑based OCR; auto‑detect language, return confidence scores, extensible via plugins.
Layout & table reconstruction	ML models (PP‑DocLayout‑V3, RT‑DETR) for reading order; table models (TATR, SLANet) produce clean Markdown tables.
Code intelligence	Tree‑sitter powered parsing of 371 programming languages – functions, classes, imports, docstrings, etc., useful for RAG pipelines.
Embeddings & search	Local ONNX embeddings or any of 165 provider APIs via liter‑llm; supports sparse, late‑interaction, and cross‑encoder reranking.
Enrichment	Built‑in NER, keyword extraction (YAKE/RAKE), summarisation, translation, redaction, page classification, QR detection, language detection, token reduction.
Schema‑driven JSON extraction	Calls a local LLM (Ollama, LM Studio, vLLM) or hosted LLM to produce structured JSON without manual prompting.
Multiple output formats	Plain text, Markdown, Djot, HTML, JSON tree, or Docling DocTags; custom renderers can be registered.
Deployment flexibility	Use as a Rust crate, Python/Node/etc. package, Docker image, Helm chart, or run the built‑in REST API (xberg serve). No GPU required for the core; optional GPU only for transcription or LLM‑backed steps.

---

How you use it

Entry point	Typical command / code snippet
CLI	xberg extract file.pdf (14 sub‑commands: extract, batch, detect, formats, serve, mcp, …). Install via Homebrew or Scoop.
Python	pip install xberg → from xberg import extract; output = extract("doc.pdf")
Rust	cargo add xberg → let out = xberg::extract(...).await?;
Node.js	npm install @xberg-io/xberg → await xberg.extract('doc.pdf')
Docker	docker run ghcr.io/xberg-io/xberg serve --port 8000 – exposes a single POST endpoint that returns JSON or Markdown.
MCP server	xberg mcp --transport stdio – provides a protocol that AI coding assistants (Claude, Cursor, Gemini, etc.) can call directly.

---

Who might benefit?

• Developers building RAG pipelines – need fast, reliable text extraction and optional embeddings from heterogeneous data sources.
• Enterprises with large document archives – can replace a patchwork of format‑specific tools with a single engine.
• AI‑assisted coding tools – the MCP mode lets agents like Claude or Cursor invoke Xberg for on‑the‑fly document parsing.
• Researchers needing reproducible preprocessing – deterministic Rust core, extensive format list, and CI‑tested across languages.

---

Quick‑start example (Rust)

use xberg::{extract, ExtractInput, ExtractionConfig};

#[tokio::main]
async fn main() -> xberg::Result<()> {
    let cfg = ExtractionConfig::default();
    let out = extract(ExtractInput::from_uri("document.pdf"), &cfg).await?;
    println!("{}", out.results[0].content);
    Ok(())
}

The same flow exists in Python, Node, Go, etc., with analogous one‑liner helpers.

---

Where to find more

• Docs: https://docs.xberg.io (installation guides, CLI reference, API docs)
• Benchmarks: https://xberg.io/benchmarks (claims fastest open‑source PDF‑to‑Markdown conversion)
• Community: Discord, Hugging Face org, and plugin marketplaces for Claude, Cursor, Gemini, etc.

---

Bottom line: Xberg is a production‑grade, cross‑language document‑intelligence toolkit that bundles format handling, OCR, layout analysis, code parsing, and optional LLM‑based enrichment—all under a single, fast Rust core. It fits squarely in the AI‑infrastructure space, especially for retrieval‑augmented generation and any workflow that needs clean, structured text from diverse sources.