delta-io/delta-rs

A native Rust library for Delta Lake, with bindings into Python

What it solves

Delta Lake is a storage format that adds reliability and performance to data lakes. It provides essential features like ACID transaction guarantees (ensuring data integrity), schema enforcement (preventing bad data), and scalable handling of large datasets. This project makes Delta Lake accessible to a wider range of developers by providing a native, high-performance implementation outside of the traditional Apache Spark ecosystem.

How it works

This project provides a native Rust library that serves as the core engine for interacting with Delta Lake tables. It offers low-level APIs for developers building custom integrations and high-level operations APIs for querying and inspecting data. Because it is written in Rust, it provides high performance and memory safety, with Python bindings that allow data scientists and engineers to use it seamlessly within their existing Python data workflows.

Who it’s for

It is designed for developers and data engineers who need to interact with Delta Lake tables using Rust or Python. It is particularly useful for those building data processing tools, frameworks, or applications that require reliable, scalable storage without relying on a full Apache Spark installation.

Highlights

  • Native Rust implementation for high performance and safety.
  • Python bindings for easy integration into data science and analysis pipelines.
  • Supports ACID transactions, schema enforcement, and scalable data operations.
  • Integrates with a wide ecosystem of tools including Polars, DuckDB, Dask, and Ray.
  • Provides both low-level APIs for developers and high-level operations for general data tasks.

Related

  • Project
    strelov1/freehire

    Open‑source job‑search platform that crawls 3.3 M+ live IT postings, deduplicates them, offers faceted search, and adds AI‑powered CV matching, fit analysis, and an in‑app assistant. Built in Go, PostgreSQL + pgvector, Meilisearch, SvelteKit front‑end, and optional LLM integration.

  • Project
    duckdb/ducklake

    DuckLake is a DuckDB extension that enables an open Lakehouse format by storing metadata in a relational catalog and data in Parquet files, supporting SQL-based updates, time travel, and schema evolution.

  • Project
    google/differential-privacy

    A collection of libraries and frameworks for generating differentially private statistics over datasets to protect individual privacy in aggregate data analysis.

  • Project
    huggingface/datatrove

    DataTrove is a Python library for processing, filtering, and deduplicating text data at very large scale. It provides modular pipeline blocks and platform-agnostic executors (local, Slurm, Ray) to efficiently prepare datasets like LLM training corpora.