google-research/weatherbench2

A benchmark for the next generation of data-driven global weather models.

What it solves

WeatherBench 2 provides a standardized framework to evaluate and compare the performance of data-driven (AI) and traditional numerical weather forecasting models. It addresses the lack of a consistent benchmark for the next generation of global weather models.

How it works

The framework consists of three main components:

  • Datasets: It provides access to publicly available, cloud-optimized ground truth and baseline datasets.
  • Evaluation Code: It offers open-source code to calculate scores. To handle large high-resolution forecast files, the code is built for scalability using Xarray-Beam and supports running on GCP DataFlow.
  • Leaderboard: A dedicated website tracks and displays the scores of state-of-the-art physical and data-driven approaches.

Who it’s for

Researchers and developers building data-driven global weather forecasting models who need a rigorous way to measure their model's accuracy against baselines.

Highlights

  • Scalable Evaluation: Designed to handle massive weather datasets using Xarray-Beam.
  • Cloud-Optimized: Integrated with GCP DataFlow for large-scale processing.
  • Community-Driven: Allows users to submit their own model results for inclusion in the benchmark.
  • Standardized Datasets: Provides a curated list of ground-truth and baseline data for fair comparison.

Related

  • Project
    benchflow-ai/benchflow

    A universal environment framework that runs AI agents against task environments and scores them through a standardized contract, treating benchmarks as frozen environments.

  • Project
    galilai-group/stable-worldmodel

    A platform for reproducible world model research and evaluation that provides a unified interface for data collection, training, and model-predictive control evaluation.

  • Project
    openml/automlbenchmark

    A framework for evaluating and comparing open-source AutoML systems using standardized environments and curated datasets to ensure reproducible research.

  • Project
    microsoft/farmvibes-ai

    A multi-modal geospatial machine learning framework for agriculture and sustainability that fuses satellite imagery, weather data, and elevation maps to generate robust agricultural insights.

  • Project
    ecmwf-lab/ai-models

    A command-line tool for running AI-based weather forecasting models, providing a unified interface for data retrieval and model execution.