open-daimax/daimax-appbench

An automated benchmark platform for AI-generated apps — built-in evaluation rules, standard sample sets, and layered test cases across a multi-dimensional metric system. One pipeline runs build, install, E2E tests, and scoring into a shareable report.

What it solves

daimax-appbench provides an automated, standardized way to evaluate the quality of AI-generated applications. It removes the need for manual testing by automating the entire pipeline—from building and installing an app to performing end-to-end (E2E) functional testing and generating a comprehensive, shareable score report.

How it works

The platform takes an app artifact (such as a URL, APK, or .app file) and runs it through a multi-stage pipeline. It uses a vision-language model to perform E2E testing, where the model "sees" the UI and executes natural-language test steps, replanning its actions if the interface behaves unexpectedly. The system scores the app across three primary dimensions: Success Rate (build/install/launch), Functional Completeness (E2E pass rate and stability), and Experience (aesthetics, performance, and size). All metrics are configurable, and the framework can be extended to support any AI coding agent by implementing a simple generator interface.

Who it’s for

This tool is designed for developers, AI researchers, and teams building AI-powered app generation platforms who need an objective, repeatable way to measure the quality of their output against standardized datasets and competitor agents.

Highlights

  • Automated E2E Pipeline: Handles the full lifecycle—build, install, test, and score—without human intervention.
  • Vision-Powered Testing: Uses vision models to navigate UIs and verify requirements, allowing for dynamic, intelligent test execution.
  • Portable Reporting: Generates a single, self-contained HTML report for every run, detailing scores across all dimensions.
  • Pluggable Architecture: Easily integrate custom AI agents or use existing ones via a simple Python interface or CLI.
  • Standardized Datasets: Includes pre-defined benchmark suites (V1 and V2) with layered test cases to ensure reproducible results.

Related

  • Project
    FrontierCS/Frontier-CS

    Frontier-CS is an open-ended, verifiable benchmark for evaluating AI models on difficult, research-grade computer science problems that go beyond traditional, saturated coding tests.

  • Project
    SlavomirDurej/claude-usage-widget

    A cross-platform desktop widget that tracks Claude.ai usage statistics, limits, and spending in real-time with visual alerts.

  • Project
    huggingface/optimum-benchmark

    Optimum‑Benchmark is a Hugging Face library for measuring inference and training performance (latency, memory, energy, throughput) of Transformers, Diffusers, PEFT, etc., across many back‑ends (PyTorch, ONNX Runtime, OpenVINO, vLLM, TensorRT‑LLM, IPEX, Llama‑Cpp, Py‑TXI) and devices (CPU, CUDA, ROCm, XPU, HPU). It provides a Python API and a Hydra‑based CLI, supports distributed launches, and outputs reproducible reports that can be pushed to the Hugging  Face Hub.

  • Project
    Kaleidophon/deep-significance

    A Python package for statistical significance testing in deep learning, providing modern tests like Almost Stochastic Order (ASO), bootstrap, permutation tests, and multiple-comparison corrections to rigorously compare model performance.