Azure-Samples/AI-Gateway

What it solves

Production-ready AI applications require more than simple API calls; they need centralized governance to manage security, reliability, observability, and cost without hindering development speed. This project provides a framework for implementing an enterprise-grade AI Gateway to mediate interactions between AI applications and the underlying models, tools, and agents.

How it works

Powered by Azure API Management (APIM), the gateway acts as a control plane that enforces policies across AI services. It utilizes Jupyter notebooks, Bicep infrastructure templates, and APIM policies to deploy specific capabilities, such as:

• Model Management: Implementing load balancing across backend pools, token rate limiting, and semantic caching using vector similarity.
• Tool Integration: Enabling secure tool access via the Model Context Protocol (MCP) and function calling with Azure Functions.
• Agent Orchestration: Managing agentic applications using frameworks like OpenAI Agents SDK and Gemini MCP Agents.
• Governance: Applying FinOps frameworks for budget control and content safety filtering for security.

Who it’s for

Enterprise developers and architects building AI applications at scale who need a centralized way to manage multiple AI models and tools while maintaining compliance, visibility, and cost control.

Highlights

• Comprehensive Lab Library: Over 30 hands-on labs covering model routing, semantic caching, and MCP protocol support.
• MCP Support: Integration with the Model Context Protocol for plug-and-play tool management.
• Enterprise Governance: Built-in support for OAuth 2.0, managed identities, and token-based metrics.
• AI-Assisted Development: Includes Copilot Agent Skills to help developers scaffold new labs and generate Bicep or Terraform configurations.