davidjurgens/potato

potato: the portable annotation tool

What it solves

Potato is a free, self-hosted annotation platform designed to handle the complex data labeling needs of NLP, AI agents, and qualitative research. It eliminates the need for coding to set up annotation tasks by using YAML configuration files, providing a unified interface for labeling everything from simple text classification to complex multi-step agent trajectories.

How it works

Users configure their annotation tasks via YAML files. The platform supports a wide array of data modalities including text, audio, video, images, and agent traces from major frameworks (like OpenAI, LangChain, and CrewAI). It integrates LLMs to provide label suggestions, uses active learning to prioritize informative samples, and offers a comprehensive suite of quality control tools such as inter-annotator agreement metrics and psychometric analysis.

Who it’s for

It is intended for AI researchers, NLP practitioners, and qualitative data analysts who need a flexible, self-hosted tool for creating high-quality datasets and evaluating AI agent performance.

Highlights

  • Multimodal Support: Labels text, audio, video, images, and documents (PDF/Word).
  • Agent Evaluation: Specialized viewers for agent traces and GUI agents with SVG overlays for clicks and scrolls.
  • AI-Powered: Includes LLM label suggestions, active learning, and a human-LLM collaborative "Solo Mode."
  • QDA Workflow: Full qualitative data analysis capabilities with living codebooks, memos, and cases.
  • Quality Control: Advanced metrics like Krippendorff's alpha, Truth Serum scoring, and behavioral tracking.
  • Deployment: Supports various authentication methods (OAuth, SSO, Clerk) and can be deployed as a HuggingFace Space.

Related

  • Project
    cvat-ai/cvat

    An open-source data annotation platform for building high-quality visual datasets for computer vision, supporting image, video, and 3D annotation.

  • Project
    inception-project/inception

    A machine-assisted text annotation platform for building annotated corpora with multi-layer schemes and knowledge-base entity linking.

  • Project
    Lapis0x0/obsidian-yolo

    An agent-native AI assistant for Obsidian that integrates RAG, tool-calling agents, and a specialized learning mode into a personal knowledge base.

  • Project
    l3p-cv/lost

    A flexible, web-based framework for collaborative image annotation that supports semi-automatic pipelines to speed up the labeling of machine learning datasets.

  • Project
    AlphaAvatar/AlphaAvatar

    A self-hostable framework for creating proactive, multimodal personal AI assistants that feature long-term memory, persona tracking, and autonomous planning.