aws/sagemaker-spark

What it solves

SageMaker Spark allows users to integrate Amazon SageMaker's machine learning capabilities directly into Apache Spark ML pipelines. It solves the problem of bridging the gap between large-scale data processing in Spark and the managed training and hosting infrastructure of SageMaker, enabling users to train and run inference on Spark DataFrames at scale.

How it works

The library provides SageMakerEstimator and SageMakerModel classes that fit into the standard Spark ML Pipeline API.

1. Training: When fit() is called on a Spark DataFrame, the library serializes the data, uploads it to S3, and triggers a SageMaker training job using either Amazon-provided algorithms (like XGBoost or K-Means) or custom algorithms in Docker containers.
2. Hosting: Once training is complete, it automatically creates a SageMaker endpoint to host the model.
3. Inference: When transform() is called, the library serializes chunks of the Spark DataFrame and sends them to the SageMaker endpoint via the InvokeEndpoint API, then appends the predictions back to the DataFrame.

Who it’s for

Data engineers and ML practitioners who use Apache Spark for data preparation and want to leverage Amazon SageMaker for scalable model training and deployment without leaving the Spark ecosystem.

Highlights

• Spark Pipeline Integration: Interleaves native Spark ML stages with SageMaker training and hosting stages.
• Flexible Algorithm Support: Works with built-in SageMaker algorithms or custom Docker-based algorithms.
• ** SBOM Support**: Supports both Scala and Python (via PySpark).
• Scalable Inference: Performs predictions on Spark DataFrames by interacting with hosted SageMaker endpoints.