StreamEval/StreamEval

🔥 The first-ever interactive streaming video generation benchmark.

What it solves

StreamEval addresses the challenge of measuring how well interactive video generation models perform in real-time. Unlike static video generation, streaming models must respond to ongoing user inputs, maintain context over time, and handle dynamic requests during sessions lasting up to 300 seconds.

How it works

The project provides a standardized testing framework that simulates interactive sessions. It evaluates models across five specific tasks—such as digital-human dialogue, world generation, and video editing—using ten distinct metrics. The system tracks how well a model follows live instructions, maintains visual stability, and preserves identity or continuity throughout the interaction.

Who it’s for

This benchmark is designed for researchers and developers building or testing generative video models that require real-time interactivity. It is suitable for those who need a rigorous, reproducible way to score model performance against a consistent set of protocols and evaluation rules.

Highlights

  • First benchmark dedicated to interactive streaming video generation.
  • Fully open-source, including all evaluation code, protocols, and scoring rules.
  • Covers five distinct tasks ranging from avatar interaction to complex video editing.
  • Includes ten specific metrics to assess quality, stability, and instruction following.
  • Monthly updates to the leaderboard to track the latest model advancements.

Related

  • Project
    TencentARC/GameHorizon

    GameHorizon is a comprehensive suite for evaluating AI models in AAA gaming environments, offering a massive dataset, an automated annotation pipeline, and a dual-track benchmark for testing multi-horizon reasoning.

  • Project
    Han-1413141/dsh-cost-meter

    A comprehensive cost-tracking plugin for the DeepSeek Harness web GUI that monitors token usage, subscription quotas, and budget limits across multiple AI providers.

  • Project
    DocAILab/XRAG

    XRAG is an open‑source benchmarking framework for Retrieval‑Augmented Generation (RAG). It lets users evaluate individual RAG components—retrievers, embeddings, LLMs, orchestrators—using classic IR metrics and LLM‑based quality scores. Features include a modular plug‑in architecture, support for OpenAI, Ollama, and Hugging Face models, a CLI, a web UI for interactive testing, and a REST API service. Install via pip, configure with a TOML file, and run benchmarks or generate synthetic QA data. Ideal for researchers and engineers building or comparing RAG systems.

  • Project
    Tencent-Hunyuan/CL-bench

    CL-bench is a benchmark suite designed to evaluate how effectively language models can learn and apply new knowledge from provided context in both professional and real-life scenarios.