microsoft/SynapseML

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

What it solves

SynapseML simplifies the creation of massively scalable machine learning pipelines. It allows users to build intelligent systems for tasks like text analytics, computer vision, and anomaly detection that can scale across single-node, multi-node, and elastically resizable clusters of computers.

How it works

Built on the Apache Spark distributed computing framework, SynapseML provides composable and distributed APIs that share the same API as the SparkML/MLLib library. This allows models to be seamlessly embedded into existing Apache Spark workflows. It abstracts over various databases, file systems, and cloud data stores to simplify data access regardless of location.

Who it’s for

Data scientists and ML engineers who use Apache Spark and work with Python, R, Scala, Java, or .NET to build large-scale ML pipelines.

Highlights

  • Vowpal Wabbit on Spark: Fast and effective text analytics.
  • The Cognitive Services for Big Data: Integration with Microsoft Cognitive Services at scale.
  • LightGBM on Spark: Distributed training of Gradient Boosted Machines.
  • Spark Serving: Ability to serve Spark computations as web services with sub-millisecond latency.
  • ONNX on Spark: Distributed and hardware-accelerated model inference.
  • Responsible AI: Tools to understand opaque-box models and measure dataset biases.
  • Isolation Forest on Spark: Distributed nonlinear outlier detection.
  • HTTP on Spark: Integration between Spark and HTTP for distributed microservice orchestration.

Related

  • Project
    JohnSnowLabs/spark-nlp

    A state-of-the-art NLP library built on Apache Spark that provides scalable, production-ready NLP annotations and pre-trained models for text, image, and speech across 200+ languages.

  • Project
    aws/sagemaker-spark

    An open-source Spark library that integrates Amazon SageMaker into Spark ML pipelines for scalable model training and inference on DataFrames.

  • Project
    databrickslabs/dbldatagen

    A Python library for generating scalable synthetic data within Databricks using Spark, designed for testing, benchmarking, and demos.

  • Project
    MAIF/shapash

    A Python library that makes machine learning models interpretable for everyone by providing user-friendly visualizations, an interactive web app, and audit reports.

  • Project
    apache/opennlp

    Apache OpenNLP is a Java library offering classic NLP tools—tokenization, POS tagging, NER, parsing, etc.—via a modular, Maven‑based architecture and both API and CLI interfaces.