dlt-hub/dlt

What it solves

dlt automates the tedious, repetitive work of moving data from various sources into structured, typed datasets. It handles the "plumbing" of data engineering—such as schema inference, handling nested data, managing credentials, and adapting to different database dialects—so you don't have to write custom code for every pipeline.

How it works

dlt is a Python library that you install directly into your existing code. You define your data source (like a REST API, SQL database, or local file) using simple Python decorators and functions. You then point that source to a destination (like DuckDB, BigQuery, or S3). dlt automatically extracts the data, normalizes it into a consistent schema, and loads it while handling schema drift and type mapping in the background.

Who it’s for

dlt is designed for developers, data engineers, and AI agents who want a lightweight, code-first way to build data pipelines. Because it is a library rather than a platform, it is ideal for those who want to keep their existing workflows and avoid "black box" systems.

Highlights

• Declarative API: Use Python decorators to define how data should be loaded, including incremental updates and merge strategies.
• Broad Compatibility: Works with 20+ destinations and thousands of data sources, including REST APIs, SQL databases, and cloud storage.
• LLM-Ready: Built from the ground up to be used by AI coding agents to generate working pipelines from prompts.
• Schema Management: Automatically infers schemas, handles nested data, and manages schema drift with configurable "contracts" to protect data quality.
• Flexible Access: Easily read loaded data back into Pandas, Polars, or Arrow, or use Ibis to perform transformations in Python that compile to SQL.