databrickslabs/dbldatagen

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

What it solves

It provides a way to generate massive amounts of synthetic data (up to billions of rows) within the Databricks environment. This is essential for testing, benchmarking, and creating demos without needing real production data.

How it works

The library uses Spark to generate data based on a specification defined in Python code. Users can either use pre-defined standard datasets or create custom specifications that define column types, value ranges, distributions, and relationships between fields. It integrates directly with Databricks Delta Live Tables and supports a plugin mechanism for third-party libraries like Faker.

Who it’s for

Data engineers and developers working within the Databricks ecosystem who need scalable, repeatable, and consistent synthetic data for their pipelines and ML feature arrays.

Highlights

  • Massive Scale: Generates billions of rows in minutes using Spark clusters.
  • Repeatable Consistency: Supports consistent primary and foreign keys for complex join and merge scenarios.
  • Flexible Generation: Supports all Spark SQL primitive types, custom distributions, and SQL expressions.
  • Schema-Driven: Can generate data to conform to an existing schema or create it ad-hoc.
  • ML Ready: Capable of generating arrays of values specifically for ML-style feature arrays.

Related

  • Project
    aws/sagemaker-spark

    An open-source Spark library that integrates Amazon SageMaker into Spark ML pipelines for scalable model training and inference on DataFrames.

  • Project
    mostly-ai/mostlyai

    A Python SDK for creating high-fidelity, privacy-safe synthetic tabular and language data using local or remote compute resources.

  • Project
    microsoft/SynapseML

    An open-source library built on Apache Spark that provides scalable, distributed APIs for machine learning tasks like text analytics, vision, and anomaly detection.

  • Project
    sdv-dev/SDV

    A Python library for creating tabular synthetic data using machine learning to emulate real-world patterns while maintaining privacy.

  • Project
    microsoft/semantic-link-labs

    A Python library for Microsoft Fabric notebooks that automates the model migration, optimization, and management of semantic models, reports, and lakehouses.