docetl: what it is, what problem it solves & why it's gaining traction

What it solves

DocETL simplifies the process of using LLMs to analyze and transform large collections of structured and unstructured data. It removes the need to manually write, wire together, and tune individual LLM calls for accuracy, cost, and latency.

How it works

Users define data processing operations (like map, reduce, and filter) using natural language prompts. DocETL orchestrates these operations, parallelizing the workload across the dataset and returning the results as queryable tables. The system automatically optimizes the pipeline by swapping models, rewriting prompts, decomposing operations, or replacing LLM tasks with code to improve accuracy and reduce costs.

Who it’s for

It is designed for developers and data analysts who need to process large-scale document collections using LLMs via a Python API, a low-code YAML configuration, or a visual interface.

Highlights

• Declarative Pipelines: Define complex workflows using simple operators like map and reduce.
• Automatic Optimization: Uses agentic rewrites to balance cost and accuracy automatically.
• Flexible Interfaces: Supports a Python API, YAML configs, and a visual playground called DocWrangler.
• Scalable Execution: Handles parallelization and rate limiting across various LLM providers.