lakesoul-io/LakeSoul

LakeSoul is an end-to-end, realtime cloud-native Lakehouse framework for fast data ingestion, concurrent updates, incremental analytics, multimodal data processing and vector search — powering next-generation BI and AI workloads.

What it solves

LakeSoul provides a production-ready lakehouse platform that goes beyond simple table formats. It eliminates the need to manually assemble separate catalogs, compaction services, and authorization layers by providing a "batteries-included" solution for managing large-scale data with ACID transactions, high-performance OLAP queries, and native support for multimodal data.

How it works

LakeSoul uses a high-performance core written in Rust for metadata management and file IO, providing consistent bindings for Java, Python, and C++. It employs an LSM-Tree like structure to handle high-write throughput for primary key tables and uses PostgreSQL for scalable metadata management and ACID control. The system supports multiple physical file formats, including Parquet and Vortex (for multimodal data and vector embeddings), and integrates with various compute engines like Spark, Flink, Ray, and Daft.

Who it’s for

It is designed for data engineers and AI practitioners who need a unified data infrastructure that supports both streaming and batch processing, as well as those training AI models using PyTorch or Ray who require efficient access to lakehouse data.

Highlights

  • Rust-Native Core: A single high-performance core ensures consistent ACID guarantees and read performance across all supported languages and engines.
  • Multimodal Support: Native processing for multimodal data and vector embeddings via the Vortex file format and integration with Daft.
  • Automated Maintenance: Built-in disaggregated multi-level compaction, table life cycle maintenance, and redundant data cleaning.
  • Broad Engine Compatibility: Supports a wide range of compute frameworks including Spark, Flink, Presto, Ray, Daft, DuckDB, and PyArrow.
  • Enterprise Security: Fine-grained RBAC and physical data isolation through an S3 proxy authorization layer.

Related

  • Project
    arkflow-rs/arkflow

    ArkFlow is a Rust‑based, high‑performance stream‑processing engine that supports many data sources (Kafka, MQTT, HTTP, files, SQL, etc.) and offers a rich set of processors, including SQL, Python UDFs, and AI‑focused components such as embedding generation, LLM chat calls, and vector similarity search. It provides durable delivery, optional exactly‑once semantics, and an optional web‑based control plane for fleet management.

  • Project
    datachain-ai/datachain

    DataChain is a Python library that turns files in cloud storage into versioned, typed datasets stored in a SQLite‑backed Dataset DB. It provides fast, SQL‑like queries, vector search, incremental processing, and an optional markdown Knowledge Base. An installed skill lets LLM agents (Claude, Cursor, Codex, etc.) read and manipulate these datasets, enabling agent‑driven data pipelines. The project also offers a cloud Studio for shared dataset registries and distributed compute.

  • Project
    OpenDCAI/DataFlow

    DataFlow is an open‑source Python framework that lets you build, run, and share low‑code pipelines for generating, cleaning, and evaluating LLM training data. It provides a library of reusable operators, a visual WebUI, an AI‑assistant for auto‑creating pipelines, and a Ray‑based distributed execution layer.

  • Project
    databricks-demos/dbdemos

    dbdemos is a pip‑installable Python toolkit that automates the setup of Databricks Lakehouse demo bundles—installing notebooks, clusters, pipelines, dashboards and optional ML models with a single command.

  • Project
    akitaonrails/ai-memory

    ai-memory is a long-term memory system for AI coding agents, storing memory as git-backed Markdown, working across agents, projects, machines, and teams, with zero-LLM default and optional LLM features.