Intercom's Strategy for Sustainable AI Advantage

Intercom successfully integrated generative AI into its customer service software by committing $100 million to replatform its business and launching Fin, an AI Agent that resolves millions of queries monthly. The company's success is attributed to a strategic shift toward an "AI-first" approach, which involved reorganizing product teams and canceling non-AI projects to ensure AI was built into the core of the product rather than added as a feature.

Building Model Fluency Through Early Experimentation

Intercom maintains a competitive edge by testing models early and often to deeply understand their limitations and opportunities. This hands-on approach allowed the company to launch Fin within four months of GPT-4's release because the team had already mapped the capabilities of GPT-3.5.

Key technical findings from their experimentation include:

Model Selection for Tasks: While Intercom initially planned a reasoning-model-based stack for "Fin Tasks" (which automates complex workflows like refunds), evaluations revealed that GPT-4.1 could handle these tasks with high reliability and lower latency.
Prompt Engineering: The team discovered that applying chain-of-thought prompting to non-reasoning queries effectively closed performance gaps.
Efficiency Gains: In Intercom's evaluations, GPT-4.1 provided the highest reliability for task completion while reducing costs by 20% compared to GPT-4o. To ensure data integrity, completeness was measured using Pass@k across five independent runs, counting a result as complete only if it succeeded in all five runs.

Accelerating Deployment with Rigorous Evaluation

Intercom utilizes a structured evaluation process to rapidly adopt new models, modalities, and architectures. This framework consists of offline tests and live A/B trials that measure instruction following, tool call accuracy, and coherence before any deployment.

Evaluation Methodology

Benchmarking: Models are tested against actual support interaction transcripts to evaluate their ability to handle multi-step instructions, maintain brand voice, and execute function calls.
A/B Testing: Live tests compare resolution rates and customer satisfaction across different models, such as GPT-4 and GPT-4.1.
Voice-Specific Metrics: For Fin Voice (powered by the Realtime API), Intercom assesses personality, tone, interruption handling, and background noise to ensure human-quality phone support.

This rigorous process enabled Intercom to migrate from GPT-4 to GPT-4.1 in a matter of days, with evaluation results completed within 48 hours of the model's release.

Maintaining Architectural Flexibility

Intercom employs a modular, model-agnostic architecture that allows the company to route queries to the most appropriate model and swap models without reengineering the entire system. This flexibility is essential for supporting multiple modalities, including chat, email, and voice, each with different latency and complexity requirements.

Architectural Evolution

Iterative Design: Fin's architecture is currently on its third major iteration, with a fourth in development, allowing the team to add complexity where necessary and simplify it as models improve.
Complexity Reduction: The high performance of GPT-4.1's instruction-following capabilities allowed Intercom to pivot its architecture for Fin Tasks, removing the need for more complex reasoning-based stacks and reducing overall latency and cost.

Expanding AI Capabilities Across the Business

By combining a modular architecture with advanced models, Intercom is extending AI utility beyond basic customer support into broader business workflows:

Support Teams: Using the Fin AI Agent to resolve the majority of inbound queries across voice, email, and chat.
Operations Teams: Utilizing Fin Tasks to automate subscription updates, account changes, and refunds.
Product Teams: Leveraging an MCP Server to allow AI tools like ChatGPT to access tickets, user data, and conversations to help refine product roadmaps and identify bugs.