nahrek/polyledger

What it solves

PolyLedger is a resumable indexer designed to collect and organize Polymarket market metadata and on-chain trade data. It solves the problem of manually aggregating fragmented data from the Polymarket CLOB and Gamma APIs and raw Polygon blockchain logs, providing a unified, queryable database for historical analysis.

How it works

The tool streams OrderFilled events from the Polygon network via Envio HyperSync and pulls market metadata from the CLOB API. It processes these data streams and writes them into a single DuckDB file. To ensure reliability, it uses atomic checkpoints (committing rows and the block cursor in one transaction) and idempotent writes to prevent duplicate data if a run is interrupted and restarted.

Who it’s for

It is built for data analysts, researchers, and traders who need a structured, SQL-queryable history of Polymarket trades and market details without managing a complex database server.

Highlights

• DuckDB Backend: Uses a serverless columnar database for efficient storage, compression, and SQL joins in a single file.
• Resumable Syncing: Built-in checkpoints allow the indexer to resume exactly where it stopped after an interruption.
• Multi-Generation Support: Decodes both V1 and V2 OrderFilled event layouts from different contract generations.
• Schema Validation: Employs Pydantic models to ensure upstream API changes trigger immediate errors rather than silent data corruption.
• Gap Recovery: Automatically flags missing token IDs and backfills them using the Gamma API.
• Parquet Export: Allows easy export of all tables to Parquet files for use with tools like pandas, polars, or Spark.