sdv-dev/SDGym
Benchmarking synthetic data generation methods.
What it solves
It provides a standardized way to compare different synthetic data generation techniques. Instead of manually testing models, users can benchmark performance, memory usage, and data quality across various modeling approaches—ranging from classical statistics to deep learning—to determine which is most effective for their specific data.
How it works
SDGym acts as a benchmarking framework that integrates with the Synthetic Data Vault (SDV) ecosystem. It allows users to select a set of synthesizers (either from the SDV library, built-in baselines, or custom-written machine learning models) and run them against a variety of datasets (publicly available or private). The framework then evaluates these models based on performance, memory consumption, and quality/privacy metrics.
Who it’s for
Data scientists and ML engineers who need to generate high-quality synthetic data and want to objectively compare different generation models to find the best balance of efficiency and accuracy.
Highlights
- Flexible Synthesizer Support: Compatible with SDV synthesizers, basic baselines, and custom user-defined training and sampling logic.
- Dataset Integration: Includes a library of public datasets and supports custom data stored locally or in Amazon S3 buckets.
- Comprehensive Metrics: Measures not just the output quality and privacy, but also the computational cost in terms of performance and memory usage.
- SDV Ecosystem: Fully integrated with the wider Synthetic Data Vault project for seamless data generation and evaluation.