deepset-ai/haystack

What it solves

Haystack is an AI orchestration framework designed to help developers build production-ready LLM applications. It addresses the complexity of moving from a simple prototype to a scalable, reliable system by providing modular components for retrieval, routing, and generation, allowing for explicit control over how data and context are managed.

How it works

Haystack uses a modular architecture where developers define pipelines and agent workflows. These pipelines can include components for indexing, retrieval, retrieval ranking, and generation. It supports native async support for streaming tokens and concurrent tool calls. The framework is model- and vendor-agnostic, meaning users can swap between different LLMs (like OpenAI, Anthropic, or local models) and infrastructure components without rewriting their entire system.

Who it’s for

Python developers building scalable RAG systems, multimodal applications, semantic search, question answering systems, and autonomous agents.

Highlights

• Production-ready Agents: Includes lifecycle hooks for guardrails, monitoring for token usage and tool calls, and a SkillToolset for progressive skill discovery.
• Context Engineering: Explicit control over how information is retrieved, ranked, filtered, and routed before reaching the model.
• Model Agnostic: Integrations with major providers like OpenAI, Mistral, Anthropic, Cohere, Hugging Face, Google, and AWS Bedrock.
• Extensible Ecosystem: A consistent interface for building and share custom components.
• Async Support: Native support for asynchronous execution and token-by-token streaming.