weecology/retriever

What it solves

Researchers and analysts often waste hours—sometimes days—getting publicly available data ready for analysis. Many datasets lack machine-readable metadata and don't follow consistent formats, so downloading, cleaning, standardizing, and importing them is tedious and error-prone. The Data Retriever automates these first steps in the data pipeline, letting you go from raw data to a usable database or file quickly.

How it works

After installing (via pip or conda, or using precompiled binaries), you run retriever update to fetch scripts for available datasets. Then you use commands like retriever install to download and load a dataset into a target format, such as MySQL, PostgreSQL, SQLite, Microsoft Access, CSV, JSON, or XML. The tool handles cleaning and standardizing the data behind the scenes. You can also download raw files with retriever download, get dataset citations with retriever citation, and create or edit your own dataset scripts using new, new_json, edit_json, and related commands. Spatial support for Postgres with PostGIS is available for vector and raster datasets.

Who it’s for

Anyone who works with public datasets for analysis—ecologists, data scientists, researchers, or analysts—who wants to skip the repetitive work of data wrangling and get straight to using the data in a database, spreadsheet, or programming language.

Highlights

• Automates downloading, cleaning, and standardizing datasets, saving hours to days of manual effort.
• Imports into multiple targets: relational databases (MySQL, PostgreSQL, SQLite, MS Access) and file formats (CSV, JSON, XML).
• Simple command-line interface with subcommands for install, download, update, and dataset script management.
• Includes a curated list of available datasets and supports custom dataset scripts.
• Optional spatial support for PostGIS to handle vector and raster data.
• Provided as a Python package (pip/conda) or standalone binaries for Windows, macOS, and Ubuntu/Debian.