lakehq/sail

What it solves

Sail is a high-performance, Rust-native replacement for Apache Spark. It eliminates the "JVM tax"—the memory overhead, garbage collection pauses, and complex tuning required by Spark—while remaining fully compatible with the Spark Connect protocol, Spark SQL, and the DataFrame API.

How it works

Sail is built using Rust, Apache Arrow, and Apache DataFusion. It uses a columnar in-memory format and SIMD instructions for vectorized execution. To maintain compatibility, it implements a custom Rust parser for Spark SQL and supports the Spark Connect protocol, allowing users to connect existing PySpark sessions to a Sail server without rewriting code. It also enables zero-copy data sharing for Python UDFs via Arrow array pointers, reducing serialization overhead.

Who it’s for

Data engineers and AI practitioners who use Apache Spark for batch and stream processing or compute-intensive AI workloads and want to reduce infrastructure costs and increase execution speed without migrating their existing codebase.

Highlights

• Drop-in Spark Compatibility: Supports Spark SQL and DataFrame API via Spark Connect, requiring no code rewrites.
• Rust-Native Performance: Eliminates JVM overhead, providing instant startup and predictable memory safety.
• Significant Cost and Speed Gains: Benchmarks show up to 10x faster execution and 98% lower infrastructure costs compared to Spark.
• Broad Integration: Native support for Delta Lake, Apache Iceberg, and various cloud storage backends (S3, Azure, GCS, etc.).
• Efficient Data Shuffling: Workers exchange Arrow columnar data directly to minimize costs for joins and aggregations.