davidjurgens/potato

What it solves

Potato is a free, self-hosted annotation platform designed to handle the complexities of modern AI development, specifically for NLP, GenAI, and AI agents. It moves beyond simple point-labeling (assigning a label to an item) by allowing researchers to capture decision boundaries, evaluate complex agent trajectories, and perform deep qualitative data analysis (QDA) without needing to write code.

How it works

Users configure their annotation projects entirely through YAML files, specifying the data types and annotation schemes they need. The platform can be deployed locally or on a server and supports a wide range of modalities including text, audio, video, images, and specialized "agent traces" (logs of LLM tool calls and reasoning chains). It integrates with LLM providers for pre-labeling and active learning, and includes a "Boundary Lab" that generates counterfactual edits to test if an annotator's label remains consistent under slight changes.

Who it’s for

It is built for NLP researchers, AI engineers evaluating agentic systems, and qualitative researchers who need a self-hosted alternative to tools like NVivo or ATLAS.ti.

Highlights

• Boundary Probing: Automatically generates minimal counterfactual edits to map decision boundaries and create contrast sets.
• Agent Evaluation: Specialized tooling for reviewing agent trajectories, including a Web Agent Viewer for screenshot-based GUI trace review.
• Comprehensive Modality Support: Handles everything from standard text classification and NER to tiered audio annotation and bounding boxes in images.
• QDA Mode: Supports living codebooks, in-vivo coding, memos, and case-based analysis for interpretive research.
• AI Integration: Features LLM-powered label suggestions, active learning to prioritize informative samples, and a collaborative "Solo Mode."