duckdb/ducklake

What it solves

DuckLake provides an open Lakehouse format that bridges the gap between structured SQL databases and file-based data storage. It allows users to maintain the benefits of a relational database—such as schema management and data updates—while keeping the actual data stored in standard Parquet files.

How it works

DuckLake separates data from metadata. It stores the metadata in a catalog database (which can be DuckDB, PostgreSQL, or SQLite) and keeps the actual data in Parquet files. By installing the DuckLake extension into DuckDB, you can attach these storage locations and interact with them using standard SQL commands as if they were a local database table.

Who it’s for

This tool is for data engineers and developers who use DuckDB and need a robust way to manage large datasets in Parquet format while requiring features like ACID transactions, schema evolution, and historical data access.

Highlights

• Supports standard SQL operations including CREATE, INSERT, and UPDATE on Parquet-backed tables.
• Enables time travel, allowing users to query previous versions of their data.
• Facilitates schema evolution through ALTER TABLE commands.
• Provides a change data feed to track modifications made to tables over time.
• Compatible with multiple catalog backends, including DuckDB, PostgreSQL, and SQLite.