facebookincubator/nimble
New and extensible file format for storage of large columnar datasets.
What it solves
Nimble addresses the limitations of existing columnar formats like Apache Parquet and ORC when handling extremely wide datasets. It is specifically designed for workloads involving thousands of columns, such as feature engineering and machine learning training tables.
How it works
Nimble uses a decoupled design that separates stream encoding from the physical layout, allowing for extensible and recursive (cascading) encodings. It utilizes Flatbuffers for efficient metadata access and employs block encoding to ensure predictable memory usage during decoding. The format is built to be SIMD and GPU friendly to leverage highly parallel hardware.
Who it’s for
It is intended for data engineers and machine learning practitioners who work with massive, wide datasets and require a high-performance, extensible storage format for feature engineering and model training.
Highlights
- Optimized for tables with thousands of columns
- Supports extensible and cascading encodings
- Uses Flatbuffers for efficient metadata management
- Designed for SIMD and GPU parallelism
- Provides a unified library to prevent implementation fragmentation