kamu-data/kamu-cli

Next-generation decentralized data lakehouse and a multi-party stream processing network

What it solves

Kamu solves the problem of global collaboration on data, which is often limited by trust, ownership, and reproducibility. It provides a way to manage dynamic structured data with a full history of changes, making it possible to verify that transformations are correct and tamper-proof. This enables organizations to share and reuse data on a scale similar to open-source software.

How it works

Kamu works as a command-line tool that ingests data from many sources, including polling web data, MQTT brokers, and blockchain logs. It stores data in the Open Data Fabric (ODF) format, which is a Web3-native format inspired by Apache Iceberg and Delta. ODF provides cryptographic integrity, stable references, and decentralized identity. Unlike Iceberg and Delta, ODF is history-preserving, meaning it keeps the full event history.

To transform data, kamu uses "derivative datasets" created via SQL queries. It integrates popular SQL engines like Flink, Spark, and DataFusion as plugins, so you can mix engines in one pipeline. Transformations are stored in metadata and are deterministic and reproducible. You can verify a dataset with the kamu verify --recursive command to confirm it matches its inputs and code.

Sharing datasets is done by pushing them to conventional or decentralized storage like S3 or IPFS. Because identity is embedded in metadata, anyone can see the owner. Kamu also supports federated querying, allowing data from different locations to be queried as one big database, with results accompanied by cryptographic commitments.

Who it’s for

Kamu is for data engineers, scientists, and analysts who need to build and manage data pipelines with strong guarantees of integrity and reproducibility. It’s also for organizations that want to share data across teams or externally without losing control or trust. Its local-first nature makes it a good fit for individuals and small teams who want a free alternative to cloud lakehouses, while still being able to scale to clusters when needed.

Highlights

  • Tamper-proof history: all changes to data are recorded with cryptographic integrity
  • Verifiable transformations: you can prove that a dataset is the exact result of its input and code
  • Local-first: runs on your laptop without accounts, scales to on-prem clusters
  • Peer-to-peer data sharing: share datasets via IPFS or conventional storage with ownership attribution
  • Federated querying: query data across locations as if it were one database
  • Integration with many SQL engines: use the best engine for each step of your pipeline

Related

  • Project
    SuYxh/ai-news-aggregator

    A TypeScript-based AI news aggregator that pulls from 14 platforms, 70+ RSS feeds, and 52 WeChat accounts, filters for AI relevance, auto-translates titles to Chinese, and provides a React web app with search, dark mode, and JSON export. Updates run automatically every two hours via GitHub Actions.

  • Project
    Project-AgML/AgML

    AgML is a Python library that provides easy access to a curated collection of public agricultural image datasets via the Hugging Face Hub. It offers unified loaders, lazy loading for huge image‑text datasets, and standard annotation schemas, enabling seamless use with the `datasets` and `transformers` ecosystems for training TensorFlow or PyTorch models.

  • Project
    MinishLab/vicinity

    Vicinity is a lightweight Python library that provides a unified, low‑dependency interface for nearest‑neighbour search over vector embeddings. It supports many back‑ends (FAISS, HNSW, Annoy, etc.), offers easy serialization (local disk or Hugging Face Hub), and includes a simple evaluation tool for measuring speed and recall.

  • Project
    VintaBytes/Ciencia-de-datos

    A free Spanish-language repository of Python data science books, Google Colab notebooks, and course materials, covering topics from basic data analysis to machine learning and deep learning. It provides hands-on, executable examples that update almost daily.