google-research/arco-era5

Recipes for reproducing Analysis-Ready & Cloud Optimized (ARCO) ERA5 datasets.

What it solves

ERA5 is a massive climate dataset (about 5 petabytes) that is hard to download and work with directly. This project removes that barrier by hosting a curated copy on Google Cloud and offering it in two usable formats: a cloud-optimized version (simply converted from the original grib format to Zarr) and an analysis-ready version (regridded and pre-processed for easy use in research and machine learning). It also gives open recipes so anyone can reproduce the datasets themselves.

How it works

The project takes data from ERA5 (a global hourly reanalysis covering 1979 to the present) and processes it in two ways. First, it converts the original grib files to Zarr, a cloud-friendly format that allows efficient access to slices of data. Second, it creates an analysis-ready version on a uniform 0.25-degree grid, combining surface and atmospheric variables and applying standard chunking. All data is stored in a public Google Cloud bucket, with folders for raw, cloud-optimized, and analysis-ready files. The datasets are updated monthly (with a 3-month delay for stable ERA5 and about a 1-week delay for preliminary ERA5T). Metadata attributes like valid_time_start and last_updated help users know exactly what time range is included. The project also includes Colab notebooks to get started quickly.

Who it's for

This is for researchers and machine learning practitioners who need large-scale climate data without the hassle of managing raw grib files. It's especially useful for those who want a ready-to-use dataset for training models (the analysis-ready version is a superset of what GraphCast and NeuralGCM were trained on) or who need fine control over interpolation and can use the cloud-optimized raw Zarr version.

Highlights

  • Two formats: cloud-optimized Zarr (for flexibility) and analysis-ready regridded data (for convenience)
  • Public and free: hosted on Google Cloud Public Datasets, with anonymous access
  • Reproducible: every step of the processing pipeline is open and documented
  • Easy to start: Colab notebooks show how to load and work with the data
  • Regular updates: monthly cadence with clear metadata for time coverage

Related

  • Project
    nimrodfisher/data-analytics-skills

    A collection of 31 portable AI skills that turn Claude or Codex into hands-on analytics partners. They cover everything from data quality checks and deep-dive analysis to dashboards and stakeholder communication—with zero setup and no company-specific configuration required.

  • Project
    microsoft/AEC-Challenge

    The ICASSP 2023 Acoustic Echo Cancellation Challenge provides researchers with large real and synthetic datasets from over 10,000 devices, along with an online subjective test framework and an objective metric service to develop and evaluate acoustic echo cancellation algorithms.

  • Project
    Desbordante/desbordante-core

    A high-performance data profiler that discovers and validates explainable, rule-based patterns in tabular, graph, and transactional data to improve data quality and ML guardrails.

  • Project
    TangleML/tangle

    A visual orchestration system for ML experiments and production pipelines that allows users to build and run containerized batch workflows via a drag-and-drop interface.