Must Haves For Agents in Production

Shipping LLM agents into production for multi-user environments requires moving beyond simple demos to a robust operational framework. Failure to implement production-grade controls often leads to leaked API keys, runaway costs from rogue agents, and undetected hallucinations across user bases.

1. Model Control

A unified layer between application code and LLM providers is essential to avoid vendor lock-in and maintain agility.

Using a single model is rarely optimal for complex agentic systems. Different tasks require different model strengths—for example, using Claude for tool calling, Gemini for multimodal tasks, or a fine-tuned open model for specific JSON outputs. A unified control layer provides several key benefits:

• Rapid Swapping: Model providers deprecate versions quickly. A control layer allows teams to swap models without rewriting core application code.
• Security: API keys are abstracted to a single secure location rather than being hard-coded.
• Configuration: Teams can manage model selection and regional configurations from a central point.

2. Prompts and Prompt Registry

Prompts should be treated as intellectual property and a second tier of code, managed via a versioned registry rather than embedded strings.

Because prompts often define the difference in performance for structured outputs, they require a professional development workflow. A prompt registry enables:

• Decoupling: Agent logic is separated from the prompt text, allowing prompt engineers to iterate without involving software developers.
• Configuration Management: The registry stores the entire configuration, including the prompt text, model selection, temperature, and attached guardrails.
• Iterative Workflow: The process moves from experimentation in a playground to saving in the registry, publishing to an agent, and running evaluations.

3. Guardrails

Input and output guardrails are mandatory to ensure compliance, security, and brand safety before any agent interacts with a user.

Guardrails should be implemented at multiple hooks: pre-LLM, post-LLM, pre-tool, and post-tool. Key focus areas include:

• Compliance: Redacting Personally Identifiable Information (PII) and Protected Health Information (PHI) to meet legal requirements.
• Input Protection: Preventing prompt injection or hacking attempts.
• Output Filtering: Ensuring agents do not use obscenities or mention competitors.

4. Budget Limiting

Strict budget caps are necessary to prevent "nightmare invoices" caused by runaway loops or rogue processes.

LLM behavior is inherently unpredictable, making it easy for a bug to trigger an infinite loop of API calls. Production systems must implement:

• Granular Caps: The ability to set daily budget limits (e.g., $1,000/day) per model or per project.
• Liability Control: Limiting the financial risk associated with multiple developers and various experimental agents.

5. Tool and MCP Server Management

Centralized authentication and granular permissions are required for the tools and Model Context Protocol (MCP) servers that agents utilize.

As agents scale to use dozens of MCP servers, APIs, and browsers, managing security becomes complex. The production approach involves:

• Centralized Auth: The agent authenticates with a gateway, and the gateway handles the downstream security and authentication for all connected tools.
• Permission Control: Implementing granular control over which agents can access specific tools, especially those that incur compute or API costs.

6. Monitoring and Tracing

Full visibility into every request, response, error, and latency spike is required to debug the "black box" nature of agents.

Without detailed traces, it is impossible to determine if a bad response was caused by a 500 error from the model, a tool providing incorrect context, or a changed API response format. Effective monitoring includes:

• User Journey Tracing: The ability to trace a single user's path through the agent's logic.
• Standardized Logging: Using OpenTelemetry-compatible traces to export data to systems like Datadog or New Relic.
• Default Observability: Utilizing a gateway that logs all traffic by default to avoid manual instrumentation of every call.

7. Evaluations (Evals)

Systematic evaluations are the only way to measure agent accuracy and catch regressions before they impact users.

Evals must be applied both before and after production deployment:

• Pre-production: Verifying that the system behaves as intended.
• Post-production: Running previous traces through new, cheaper models to test viability, or detecting when a percentage of queries start failing over time.
• Component Testing: Evaluating both the entire system and individual components to identify whether a prompt or a tool needs updating.