spotify/featran

What it solves

Featran simplifies the time-consuming process of feature engineering for data science and machine learning. It replaces naive, manual implementations of feature transformations (like scaling and encoding) with a structured, type-safe approach, making complex feature sets more manageable.

How it works

The library uses a two-step process for feature transformation: a global aggregation step to summarize the data (e.g., finding the minimum and maximum values) followed by an element-wise mapping step to transform the data based on those summaries. Users define a FeatureSpec to specify which fields should be transformed and which transformers (such as MinMaxScaler or OneHotEncoder) to apply.

Who it’s for

It is designed for data scientists and machine learning engineers working in the Scala ecosystem who need to actually transform raw data into features for ML models.

Highlights

• Broad Integration: Supports extracting data from Scala collections, Apache Flink, Scalding, Scio, and Apache Spark.
• Flexible Output: Can export features as Scala collections, Breeze vectors, TensorFlow Example Protobufs, XGBoost LabeledPoints, and NumPy files.
• Training/Test Split Support: Allows importing aggregations from previous extractions to ensure consistency across training, validation, and test sets.
• Composable Specs: Supports composing feature specifications and separating outputs.