Building Reliable Agentic AI Systems: The PRINCE Case Study

Executive Summary

Bayer's PRINCE (Preclinical Information Center) platform demonstrates that reliability in agentic AI systems is achieved not through better prompts alone, but through the combination of context engineering (controlling what information a model sees) and harness engineering (building a robust orchestration layer around the model). By evolving from a simple search tool to an active research assistant, PRINCE uses a multi-agent workflow to navigate complex preclinical data silos, combining Retrieval-Augmented Generation (RAG) for unstructured PDFs and Text-to-SQL for structured metadata.

The Evolution of PRINCE: From Search to Do

PRINCE was developed in three strategic phases to address the challenges of fragmented data silos and the limitations of keyword-based search in preclinical research:

  1. Search: A unified gateway consolidating structured study metadata from disparate in-house silos.
  2. Ask: The introduction of RAG, allowing researchers to query unstructured data (such as scanned PDF reports) using natural language.
  3. Do: The current agentic phase, where multi-agent systems orchestrate complex workflows, such as drafting regulatory documents.

Technical Architecture and Orchestration

PRINCE is built using LangGraph for orchestration and FastAPI for the backend, utilizing a conversational UI built with React. The system is designed to avoid treating the prompt as a single large container, instead implementing a strict "context discipline" where different agents receive only the specific context required for their task.

The Multi-Agent Workflow

The system coordinates specialized agents to ensure accuracy and verifiability:

  • Clarify User Intent: A "fail-fast" mechanism that proactively asks clarifying questions to pinpoint the domain or data type, preventing wasted execution on vague queries.
  • Think & Plan (Process Reflection): A dedicated reasoning space where the system evaluates its own trajectory. This step is critical for selecting the correct tool when multiple tools have overlapping domain boundaries.
  • Researcher Agent: A hybrid retriever that routes queries to either a RAG pipeline for unstructured PDFs or a Text-to-SQL pipeline for structured data in Amazon Athena.
  • Reflection Agent (Data Reflection): Unlike the Think & Plan step, this agent evaluates whether the retrieved data is sufficient to answer the query. If gaps are found, it generates follow-up questions for the Researcher Agent.
  • Writer Agent: Synthesizes the final answer, ensuring every claim is grounded in the provided context with granular citations linking back to specific document chunks and page numbers.

Deep Dive: Retrieval Strategies

Hybrid RAG for Unstructured Data

To handle decades of complex PDF reports, PRINCE employs a sophisticated query-time pipeline:

  1. Keyword Extraction & Metadata Filtering: LLMs extract keywords and generate metadata filters (e.g., study_id) to narrow the search space in Amazon OpenSearch.
  2. Query Expansion: A smaller model generates five semantically similar queries to account for terminology variations.
  3. Weighted Hybrid Search: The system combines semantic vector similarity (kNN) and keyword search with a 0.7/0.3 weight distribution.
  4. Reranking: A cross-encoder model (bge-reranker-large) refines the top 20 results down to the 7 most relevant chunks.

Text-to-SQL for Structured Data

For quantitative queries, PRINCE converts natural language to Athena SQL using:

  • Dynamic Few-Shot Prompting: Relevant SQL translation examples are retrieved from a vector database and injected into the prompt.
  • Iterative Error Correction: If a query fails, the database error is fed back to the LLM, which attempts to correct the SQL up to three times.

Engineering for Production Reliability

Reliability is managed through "harness engineering," which provides a control layer around the non-deterministic nature of LLMs.

Resilience and Recovery

  • State Persistence: Workflow state is persisted in PostgreSQL (via LangGraph checkpointers) and DynamoDB, allowing the system to resume from the exact node where a failure occurred.
  • LLM Fallbacks: If a primary model fails after several retries, the system automatically switches to an alternative provider to ensure continuity.
  • User-Initiated Retries: Users can manually retry failed queries, skipping successfully completed steps via the persisted state.

Trust and Evaluation

  • Transparency: The UI displays intermediate reasoning steps and the specific tools utilized, allowing researchers to follow the system's logic.
  • Evaluation Framework: The system uses the RAGAS framework for both dataset evaluations (against expert-curated ground truth) and daily live traffic evaluations to monitor for hallucinations.

Critical Analysis and Community Insights

While the technical architecture is comprehensive, community discussion highlights several critical tensions in building such systems:

  • The "Vibe-y" Nature of Agent Decomposition: Some critics argue that decomposing a system into "Researcher," "Writer," and "Reflection" agents may be more of a satisfying flowchart than a proven optimization, suggesting that a single frontier model with a large context window might achieve similar results more simply.
  • Data Quality vs. Agent Tuning: Industry practitioners note that the ratio of effort spent on data cleaning versus agent tuning is often closer to 99/1, emphasizing that the agent is only as good as the underlying database.
  • The Risk of Hallucinations in High-Stakes Domains: In scientific research, the cost of a hallucination is extremely high. Some observers point to the reported average user satisfaction score (3.1/5.0) in the associated research paper as evidence that agentic RAG still has significant gaps in meeting professional scientific needs.

"Reliability comes from engineering both the context the model sees and the harness within which the model acts." — Bayer Engineering Team

Sources