mostly-ai/mostlyai

Synthetic Data SDK ✨

What it solves

It provides a way to create high-fidelity, privacy-safe synthetic versions of existing datasets. This allows organizations to use data for testing, analysis, or training without exposing sensitive original information, while maintaining the statistical properties of the source data.

How it works

The SDK allows users to train "Generators" on tabular or language data. These generators can then be used to produce synthetic datasets or probe for specific samples. It supports two operational modes: LOCAL (training and generation on the user's own compute) and CLIENT (connecting to a remote endpoint).

Technically, it employs several model types depending on the data:

  • Tabular data: Uses TabularARGN for high efficiency and quality.
  • Graph relations: Uses DNN-based match making.
  • Text data: Uses either a basic LSTM from scratch or fine-tuned language models from the Hugging Face Hub.

Who it’s for

Data scientists, ML engineers, and privacy officers who need to generate realistic synthetic data for development or sharing purposes while adhering to privacy constraints.

Highlights

  • Broad Data Support: Handles mixed-type data (categorical, numerical, geospatial, text), multi-table structures, and time-series.
  • Privacy Controls: Includes support for Differential Privacy and quality metrics for privacy and fidelity.
  • Flexible Sampling: Supports up-sampling, conditional simulations (generating data based on specific column values), and re-balancing underrepresented segments.
  • Automated QA: Generates in-depth HTML reports for visual analysis of data quality.
  • Integration: Includes connectors for various databases (PostgreSQL, MySQL, Oracle, MS SQL) and cloud storage (Google BigQuery).

Related

  • Project
    hitsz-ids/synthetic-data-generator

    A specialized framework for generating high-quality, privacy-preserving synthetic tabular data using statistical models, GANs, and LLMs.

  • Project
    sdv-dev/SDV

    SDV (Synthetic Data Vault) is a production‑ready Python library for generating realistic tabular synthetic data. It offers a range of ML models (copulas, GANs), handles preprocessing, anonymisation, and relational constraints, and provides built‑in evaluation and visualisation tools.

  • Project
    Data-Centric-AI-Community/fg-data-synthetic

    A package for generating synthetic tabular and time-series data using generative models like GANs and Gaussian Mixtures to ensure privacy and augment datasets.

  • Project
    databrickslabs/dbldatagen

    A Python library for generating scalable synthetic data within Databricks using Spark, designed for testing, benchmarking, and demos.

  • Project
    tabularis-ai/be_great

    A framework that uses pretrained Transformer language models to generate high-quality synthetic tabular data, supporting both training-based and schema-based generation.