Streambed: Stream Postgres to Iceberg on S3

The challenge of bridging the gap between operational databases and analytical environments is a perennial struggle for data engineers. Traditionally, the path from a production Postgres database to a business intelligence (BI) dashboard is fraught with complexity: multiple read replicas, bespoke ETL pipelines, and the expensive overhead of maintaining a analytical warehouse.

Streambed emerges as a solution to this complexity by streamlining the movement of data from Postgres to Apache Iceberg on S3. By leveraging logical replication, it allows organizations to treat their object storage as a high-performance analytical layer, reducing the number of components in the data stack.

The Architecture: From WAL to Iceberg

Streambed's core value proposition is the reduction of architectural overhead. Instead of relying on a heavy ETL framework, Streambed connects to Postgres as a logical replication subscriber. This is the same mechanism used by Postgres read replicas, allowing it to capture changes in real-time via the Write-Ahead Log (WAL).

Once captured, these changes are streamed directly into Apache Iceberg tables on S3. This approach transforms the operational data into a standardized, open table format that is highly optimized for analytical queries. To make this data accessible, Streambed provides an embedded DuckDB instance, allowing users to query their Iceberg tables using the Postgres wire protocol (psql). This creates a seamless experience where the data is stored on S3 but feels like a Postgres-compatible database.

Key Advantages of the Open Table Format

By targeting Apache Iceberg, Streambed provides several critical advantages over traditional CSV or JSON dumps to S3:

• Separation of Compute and Storage: As noted by community members, this architecture allows for significant cost reductions by separating the query server from the storage layer (S3).
• Open Standards: Using Iceberg prevents vendor lock-in and allows the data to be queried by various other query engines that support the Iceberg specification.
• Reduced Component Count: The author, a former tech lead for the Postgres team at Cloudflare, designed Streambed to answer the question: "What's the fewest components I could get away with?"

Critical Considerations and Technical Hurdles

While the streamlined approach is promising, the community discussion highlights several technical challenges that must be considered when implementing such a system.

The ELT vs. ETL Debate

One of the primary critiques is that while Streambed simplifies the "Extract and Load" portions of the pipeline, it does not eliminate the need for "Transform." For realistic analytics, raw replicated data often needs to be transformed into a star schema or other analytical models.

"Replicating the Postgres WAL to S3 and Iceberg reliably is a hard problem but it’s not accurate to say that no ETL is needed here... anyone who’s interested into using this for realistic analytics they will have to transform the data at some point."

CDC Implementation Details

Change Data Capture (CDC) is notoriously difficult to get right. Specifically, the use of TOAST columns in Postgres can create issues. If a table has large values that are not changed in an update, the WAL records may not include the full values unless REPLICA IDENTITY FULL is configured. This is a critical configuration detail that users must be aware of to ensure data integrity in the downstream Iceberg tables.

Performance and Pushdown

Comparing Streambed to other similar projects like pg_lake or DuckLake, users have pointed out the importance of "pushdown capabilities." Without efficient pushdown, OLAP queries can become prohibitively expensive and expensive in terms of latency, as the same amount of data must be transferred from S3 to the compute engine.

Conclusion

Streambed is an early-stage project that aims to solve a long-standing pain point in the data engineering world: the overhead of maintaining the data lakehouse architecture. By converting Postgres WAL changes into Iceberg tables on S3, it provides a lightweight, accessible way to run analytical queries without the need for a massive ETL infrastructure. However, as with any other data pipeline, the data modeling and transformation layer remains a critical piece of the puzzle for any organization scaling their analytics.