TheDesignFounder/DreamLayer
Benchmark diffusion models faster. Automate evals, seeds, and metrics for reproducible results.
What it solves
DreamLayer AI provides a structured environment for benchmarking and evaluating image and video diffusion models. It eliminates the manual effort of tracking prompts, seeds, and configurations across different models, automating the process of generating outputs and calculating quality metrics to ensure reproducible and consistent comparisons.
How it works
The platform consists of a React frontend and Flask-based backend services that integrate with ComfyUI for local image workflows and various external APIs (such as OpenAI, Flux, and Ideogram) for closed-source models. It uses an SQLite database to log every run's metadata—including prompts, seeds, samplers, and model hashes—and stores computed metrics in dedicated tables. Metrics are calculated either live during generation or on-demand via API calls.
Who it’s for
- AI researchers and ML engineers who need to compare diffusion models and evaluate generation quality.
- Labs and teams building internal benchmarking pipelines for generative AI.
- Open-source model creators testing checkpoints, LoRAs, and workflows.
- Developers wanting to integrate custom evaluation metrics into their workflows.
Highlights
- Automated Benchmarking: Performs sweeps across multiple prompts, seeds, and samplers to reduce manual evaluation time.
- Comprehensive Metrics: Built-in support for image metrics (CLIPScore, FID, LAION aesthetic, YOLOv8 composition) and video metrics (FVD, SSIM, PSNR, LPIPS, temporal flickering, and motion smoothness).
- Reproducibility: Every run is persisted to SQLite, allowing any specific run to be replayed via its ID.
- Flexible Model Support: Works with local open-source checkpoints (.safetensors, .ckpt) and closed-source APIs.
- Researcher-Friendly Exports: Supports exporting results to CSV or ZIP report bundles for paper appendices or leaderboard submissions.