TheDesignFounder/DreamLayer-Eval

What it solves

DreamLayer Eval provides a structured way to benchmark and compare image and video diffusion models. It eliminates the manual effort of tracking prompts, seeds, and configurations by automating the generation process and calculating quality metrics in a reproducible, logged environment.

How it works

The platform operates as a local workbench with a React frontend and Flask-based backend services. It integrates with ComfyUI for image workflows and supports both local open-source checkpoints (via .safetensors/.ckpt files) and closed-source API models (such as DALL·E, Flux, and Ideogram). All generation metadata—including seeds, samplers, and model hashes—and computed metrics are stored in a SQLite database, allowing users to replay specific runs or compare different runs side-by-side.

Who it’s for

• AI researchers and ML engineers evaluating the quality of image and video generation models.
• Labs and teams building internal benchmarking pipelines.
• Open-source model creators testing new checkpoints or LoRAs.
• Developers who need to integrate custom evaluation metrics into their workflows.

Highlights

• Automated Benchmarking: Performs sweeps across multiple prompts, seeds, and samplers to reduce manual testing time.
• Comprehensive Metrics: Includes built-in support for CLIPScore, FID, LAION aesthetic scoring, and YOLOv8-based composition analysis for images, as well as FVD, SSIM, PSNR, and temporal consistency metrics for video.
• Reproducibility: Every run is persisted to SQLite, ensuring that any result can be replayed using a run_id.
• Flexible Model Support: Works with local GPU setups (CUDA, MPS, CPU) and external APIs.
• Researcher-Friendly Exports: Supports exporting results to CSV or ZIP report bundles for use in papers or leaderboards.