ICIJ/datashare

A self‑hosted search engine for documents

What it solves

Datashare provides a secure, self-hosted way for journalists and researchers to search and analyze massive amounts of heterogeneous data. It eliminates the need for external cloud services, allowing users to maintain full control over sensitive materials while processing millions of documents.

How it works

The platform ingests various file types (PDFs, emails, spreadsheets, images, and archives), extracts text—using OCR for scans and images—and enriches the data with metadata and named-entity extraction. It indexes this information using Elasticsearch and stores it in PostgreSQL, exposing the results through a REST API and a search user interface.

Who it’s for

It is designed for investigative journalists, solo reporters, and small to large newsrooms that need to handle large-scale document leaks or archives securely.

Highlights

  • Full-text search: Supports PDFs, emails, office documents, images, and archives.
  • OCR integration: Converts visual text in images and scans into searchable text.
  • Named-entity extraction: Automatically detects people, organizations, locations, and email addresses.
  • Advanced Querying: Features boolean, wildcard, and fuzzy queries combined with facets.
  • Collaborative Tools: Includes a team/server mode for shared tags and recommendations.
  • Extensible: Built with a plugin architecture for custom modules.

Related

  • Project
    codelibs/fess

    Fess is an enterprise search server that crawls websites, file systems, and data stores, then provides full-text search through a browser-based admin UI and REST API. It runs on Java and uses OpenSearch underneath, so you don't need to manage the search engine directly.

  • Project
    readur/readur

    A high-performance document management system with integrated OCR and full-text search for organizing and accessing digital documents.

  • Project
    elastic/elasticsearch

    A distributed search and analytics engine and vector database that enables near real-time search and analysis over massive datasets for AI and monitoring applications.

  • Project
    datahub-project/datahub

    An open-source metadata management platform that turns enterprise data into trusted context for humans and AI agents through discovery, governance, and observability.

  • Project
    linuxrebel/DocuBrowser

    DocuBrowse is a cross‑platform desktop/web app that indexes a wide range of document types and lets you search them by keywords and by AI‑generated semantic similarity, all using locally‑run Ollama models. It offers deep‑link passage navigation, on‑demand AI summaries, PII filtering, and runs entirely offline.