Snowflake-Labs/pg_lake

pg_lake: Postgres with Iceberg and data lake access

What it solves

pg_lake allows you to use PostgreSQL as a complete lakehouse system. It bridges the gap between traditional database tables and modern data lake formats, enabling you to store, query, and modify data residing in object storage (like S3) using standard SQL, while maintaining transactional integrity.

How it works

It uses a modular architecture consisting of PostgreSQL extensions and a separate process called pgduck_server. When you run queries, the system orchestrates the work between PostgreSQL and DuckDB. PostgreSQL handles the metadata, transactions, and SQL planning, while the high-performance DuckDB engine executes the heavy lifting for data scanning and computation. This separation allows for fast, parallelized queries on external files without compromising the stability of your Postgres instance.

Who it’s for

This project is for data engineers and developers who want to work with large-scale data in formats like Iceberg, Parquet, or JSON without leaving the familiar PostgreSQL environment. It is ideal for teams that need to combine local database tables with external data lake files in a single, transactional SQL interface.

Highlights

  • Full support for creating and modifying Apache Iceberg tables directly from SQL.
  • Seamless integration with object storage (S3) for importing and exporting data in Parquet, CSV, and JSON formats.
  • Ability to combine local heap tables, Iceberg tables, and raw files in the same query.
  • Automatic schema inference for external data sources.
  • High-performance query execution powered by the DuckDB engine.
  • Support for geospatial formats via GDAL and a built-in map type for semi-structured data.

Related

  • Project
    drmingler/docling-api

    A scalable backend server that uses IBM's Docling to convert various document formats like PDF and DOCX into Markdown via a FastAPI and Celery-based API.

  • Project
    skrub-data/skrub

    skrub is a Python library that supplies dataframe‑focused cleaning, encoding, and similarity‑grouping tools, all wrapped as scikit‑learn‑compatible transformers, to streamline the preprocessing stage of tabular machine‑learning pipelines.

  • Project
    chrismattmann/tika-python

    A Python port of Apache Tika that enables text and metadata extraction, MIME type detection, and language identification from various file formats via a REST server.

  • Project
    steipete/birdclaw

    A local-first tool that imports Twitter/X archives into a SQLite database for private, searchable access to tweets, DMs, and follow graphs via a web app and CLI.