awslabs/open-data-registry

A registry of publicly available datasets on AWS

What it solves

Sharing large datasets in the cloud can be difficult for both providers and users. This project simplifies the process by providing a centralized, searchable registry for publicly available datasets hosted on AWS. It helps users spend less time acquiring data and more time performing analysis using AWS tools like Amazon Athena, EC2, or Lambda.

How it works

Datasets are registered by creating a YAML metadata file within the repository. This file acts as the source of truth, containing details such as the dataset name, description, contact information, licensing, and specific AWS resource locations (like S3 buckets or SNS topics). The registry uses these files to automatically populate the official Registry of Open Data on AWS website and provide structured listings for discovery.

Who it’s for

This project is for data providers—such as government organizations, researchers, and businesses—who want to make their datasets discoverable to the AWS community. It is also for data scientists and developers looking to find and utilize high-quality, publicly available data for their own cloud-based projects.

Highlights

  • Centralized discovery for diverse datasets hosted on AWS infrastructure.
  • Standardized metadata format ensures consistency across all entries.
  • Includes "Data at Work" sections to showcase tutorials, tools, and publications that demonstrate how to use the data.
  • Enables community contributions via pull requests for new datasets or usage examples.

Related

  • Project
    ModernRelay/omnigraph

    Omnigraph is an open‑source graph database built on the Lance columnar format that runs on any S3‑compatible object store. It offers typed schemas, multimodal retrieval (graph + vector + full‑text), Git‑style branching, and Cedar‑based security policies, all managed via Terraform‑like `cluster` files. Designed for long‑horizon AI agents, it lets multiple agents share a versioned world model and retrieve context in a single query.

  • Project
    HelixDB/helix-foundry

    A local-first data workspace that connects multiple company data sources into a unified ontology for private, AI-powered exploration and querying.

  • Project
    simonlin1212/global-stock-data

    A Python‑only skill that lets AI coding assistants fetch US‑centric market data (quotes, K‑lines, fundamentals, options Greeks, short‑interest, SEC filings, macro rates, FX, Taiwan stocks, energy data) from 17 public sources. It provides 50+ endpoints across 16 logical layers, requires only the `requests` library, and includes a compliance tier table to guide commercial use.

  • Project
    supervc-stack/VectorChord

    A PostgreSQL extension for high-performance, cost-effective vector search that enables billion-scale vector hosting on modest hardware using RaBitQ compression.