lakesoul-io/LakeSoul

LakeSoul is an end-to-end, realtime cloud-native Lakehouse framework for fast data ingestion, concurrent updates, incremental analytics, multimodal data processing and vector search — powering next-generation BI and AI workloads.

LakeSoul – A Production‑Ready Lakehouse Platform

LakeSoul is an open‑source lakehouse framework that goes beyond just a table format (like Apache Iceberg). It provides a single, Rust‑implemented core for metadata management and file I/O, with idiomatic bindings for Java, Python, C++, and Scala. The goal is to give data‑engineers and data‑scientists a batteries‑included solution that works the same way whether you are reading data with Spark, streaming with Flink, querying with Presto, or training models with PyTorch, Ray, or Daft.

What it actually does

Feature Why it matters
ACID transactions & MVCC Guarantees consistent reads/writes across batch and streaming workloads.
Incremental upserts (row & column) Enables high‑throughput writes on primary‑key tables without costly full rewrites.
Multi‑engine support Spark 3.5, Flink 1.20, Presto Velox, Ray 2.55, Daft 0.7+, DuckDB, PyArrow, Pandas – all can read/write the same tables with identical semantics.
Rust‑native metadata & I/O layer Single high‑performance implementation, avoiding divergent code paths per language.
Automatic, disaggregated multi‑level compaction Keeps storage layout optimal without manual admin work.
Fine‑grained RBAC + S3 proxy auth Secure, per‑table or per‑workspace access control, enforced both in metadata (PostgreSQL) and object storage.
Vector & multimodal file format (Vortex) Stores embeddings or other high‑dimensional data alongside regular rows.
Built‑in CDC & exactly‑once streaming Real‑time ingestion from sources like MySQL with automatic schema sync.
Snapshot (time‑travel) & rollback Easy point‑in‑time queries and recovery.

Typical use cases

  • Real‑time data warehousing – ingest change‑data‑capture streams, keep tables up‑to‑date with exactly‑once guarantees, and run incremental analytics.
  • AI/ML data pipelines – read training data directly from the lakehouse with the native Python reader (no Spark needed) or as a PyTorch Dataset; store embeddings in the Vortex format for later vector search.
  • Multi‑engine analytics – run ad‑hoc SQL in Presto/Trino, batch jobs in Spark, or low‑latency queries in DuckDB, all against the same tables.
  • Secure multi‑tenant environments – separate workspaces and enforce row‑level security via PostgreSQL RBAC and an S3 proxy.

Ecosystem & Integration

  • Compute engines: Spark, Flink, Presto (Velox), Ray, Daft, DuckDB, Pandas, PyArrow.
  • Storage back‑ends: HDFS, Amazon S3 (and any S3‑compatible object store).
  • File formats: Default vortex‑compact, Apache Parquet, and the open‑source Vortex format for multimodal data.
  • Metadata store: PostgreSQL (used for ACID, MVCC, and RBAC).
  • Language bindings: Java/Scala, Python, C++.

Project health

  • License: Apache‑2.0 (permissive, commercial‑friendly).
  • Governance: Donated to the Linux Foundation AI & Data (LF AI & Data) sandbox project (May 2023).
  • CI/CD: Automated Maven, Flink CDC, native‑build, and Python tests run on every PR; badges show passing status.
  • Community: Discord channel, mailing list, and contribution guidelines are provided. The repository includes extensive tutorials (Python AI training, Flink CDC sync, snapshot management, etc.).
  • Roadmap: Active development toward newer Spark/Flink versions, vector ANN search, deeper integration with Gluten/Velox, and performance enhancements.

Quick start

The docs offer a one‑click local environment setup and a collection of example notebooks/scripts that demonstrate:

  1. Loading data with the native Python reader.
  2. Streaming a MySQL CDC feed into LakeSoul via Flink.
  3. Training a PyTorch model on data stored in LakeSoul.
  4. Performing time‑travel queries in Spark.

Bottom line: LakeSoul is a genuine, production‑grade lakehouse platform that unifies batch, streaming, and AI workloads under a single, Rust‑based core. It is well‑suited for teams that need ACID guarantees, multi‑engine compatibility, and built‑in security without stitching together separate catalog, compaction, and auth services.

Related

  • Project
    arkflow-rs/arkflow

    High‑performance Rust stream processing engine that integrates AI capabilities for real‑time data processing, inference, anomaly detection, and complex event processing.

  • Project
    datahub-project/datahub

    An open-source AI data catalog and metadata platform that enables data discovery, governance, and observability across fragmented data ecosystems.

  • Project
    lakekeeper/lakekeeper

    A secure, Rust-based Apache Iceberg REST Catalog that centralizes access control and table management across multiple compute engines.

  • Project
    lance-format/lance

    An open lakehouse format for multimodal AI that provides high-performance vector search, random access, and native storage for images, video, and audio.

  • Project
    rocketride-org/rocketride-server

    An open-source AI Development Environment (AIDE) that lets developers visually build, debug, and deploy high-performance AI pipelines using a C++ runtime.