facebookincubator/nimble

What it solves

Nimble is a columnar file format designed to replace existing formats like Apache Parquet and ORC, specifically for large datasets. It addresses the inefficiencies of current formats when dealing with "wide" tables—those containing thousands of columns—which are common in machine learning training tables and feature engineering workloads.

How it works

Nimble decouples stream encoding from the physical layout of the file, allowing for extensible and recursive (cascading) encodings. It utilizes Flatbuffers for efficient metadata access and employs block encoding instead of stream encoding to ensure predictable memory usage during reading and decoding. The system is designed to be SIMD and GPU-friendly to leverage highly parallel hardware.

Who it’s for

It is intended for developers and data engineers working with massive datasets, particularly those involved in ML training and feature engineering where wide-table structures are prevalent.

Highlights

• Optimized for Wide Tables: Efficiently supports thousands to tens of thousands of columns.
• Extensible Encoding: Allows users to add new encodings or apply them recursively.
• Hardware Acceleration: Designed for high parallelism using SIMD and GPU-friendly encodings.
• Efficient Metadata: Uses Flatbuffers to reduce overhead when accessing large metadata sections.
• Unified Implementation: Provided as a single library to prevent fragmentation across different environments.