delta-io/delta

What it is

Delta Lake is an open-source storage framework that lets you build a Lakehouse architecture — a system that combines the low-cost storage of a data lake with the reliability and transactional features of a data warehouse. It works with popular data processing engines like Apache Spark, PrestoDB, Flink, Trino, and Hive, and offers APIs for Scala, Java, Rust, Ruby, and Python.

This repository (delta-io/delta) is the core project, but it's part of a larger family of Delta Lake repositories (like delta-rs, delta-sharing, and kafka-delta-ingest) under the delta.io organization.

Key capabilities

• ACID transactions — Delta Lake ensures that concurrent reads and writes are serializable (i.e., the result is as if operations happened one at a time, in some order). This is achieved through a transaction log that records every change.
• Multiple connectors — The README highlights integrations with:
  • Apache Spark (read and write)
  • Apache Flink (write, preview)
  • PrestoDB (read)
  • Trino (read and write)
  • Delta Standalone (a single-node Java library for Scala/Java projects, including Flink, Hive, Beam, and PrestoDB)
  • Apache Hive (read)
  • Delta Rust API (low-level access for Rust, with Python and Ruby bindings)

How it works (the transaction protocol)

Delta Lake's guarantees come from a Delta Transaction Log Protocol (specified in the PROTOCOL.md file). The protocol defines how table metadata is stored in a transaction log, and it's what gives the format its transactional behavior. The Delta Standalone library implements this protocol to let non-Spark tools read and write Delta tables.

Compatibility

• API compatibility: The documented public APIs (Scala, Java, Python) are stable. Spark-based APIs (like spark.read and df.write) remain stable within a major release (e.g., 1.x.x). Internal classes may change.
• Data storage compatibility: Newer versions of Delta Lake can always read tables written by older versions (backward compatible). However, older versions may not be able to read tables written by newer versions (forward compatibility is not guaranteed) — this happens when new features are added to the transaction protocol, signaled by incrementing the minimum reader/writer version.

Requirements for underlying storage

Delta Lake's ACID guarantees depend on the storage system providing three things:

1. Atomic visibility — a file is either fully visible or not visible at all.
2. Mutual exclusion — only one writer can create or rename a file at its final destination.
3. Consistent listing — once a file is written, all future directory listings must include it.

Building and developing

• The project is compiled with SBT (Scala Build Tool).
• For building, you need Java 17 or newer (check with java -version).
• Common commands: build/sbt compile, build/sbt package, build/sbt test.
• For Python tests, the README describes setting up a Conda environment (with JDK 11 for that specific build path) and running python/run-tests.py.
• For IntelliJ development, the README recommends JDK 11 and provides detailed setup steps (including troubleshooting for generated parser files).

Community and license

• License: Apache License 2.0.
• Community channels: Slack (register at go.delta.io/slack), LinkedIn, YouTube, and a public mailing list.
• Contributions are welcome, and the project follows a Code of Conduct.

Where to go next

The README points to the Delta Lake Documentation for full details, a Quick Start Guide for Scala/Java/Python, and a roadmap for future plans.