combust/mleap

What it solves

Deploying machine learning pipelines is often time-consuming and difficult because pipelines trained in frameworks like Apache Spark or Scikit-learn typically require their full, heavy dependencies (like a SparkContext or pandas) to run in production. MLeap provides a way to export these pipelines into a portable, lightweight format that can be executed without those original training-time dependencies.

How it works

MLeap introduces a serialization format called Bundle.ML and a performant execution engine implemented in Scala. Data scientists can train a pipeline in Spark or Scikit-learn, export it once using MLeap's serialization tools, and then run it on the lightweight MLeap Runtime on the JVM. This removes the dependency on the original training frameworks during the scoring phase.

Who it’s for

Data scientists and engineers who need to deploy ML pipelines from Spark or Scikit-learn to production environments where a lightweight, high-performance execution engine is required.

Highlights

• Framework Agnostic Execution: Run pipelines trained in Spark or Scikit-learn on the JVM without needing Spark, sklearn, numpy, or pandas.
• Portable Formats: Supports JSON and Protobuf for serialization.
• Parity Testing: Includes full parity tests to ensure consistency between Spark and MLeap pipelines.
• Extensible: Allows for the implementation of custom transformers and data types.
• Broad Compatibility: Supports various versions of Spark, Scala, Java, and Python, including integrations with XGBoost and TensorFlow.