Ada Customer Service Automation with GPT-4

Ada has transitioned its AI-native customer service platform to a reasoning-based architecture powered by OpenAI's GPT-4, resulting in a doubling of automated resolution rates. By shifting the primary performance metric from "containment rate" to "resolution rate," Ada has improved the actual customer experience while maintaining high automation levels.

Shifting Metrics from Containment to Resolution

Ada has introduced "resolution rate" as a superior metric to the industry-standard "containment rate." While containment rate only measures whether a customer inquiry was handled by a chatbot without reaching a human agent, it does not account for whether the customer's problem was actually solved.

To implement this, Ada developed an evaluation framework using GPT-4 and historical data to assess conversations based on whether customers received relevant, accurate, and safe replies. This automated evaluation system achieves 80–90% agreement with human reviewers.

Technical Implementation and Model Selection

Ada utilizes a multi-agent configuration powered by OpenAI's API, consisting of a central planning agent and several subagents. Every customer query undergoes multiple turns of processing—including understanding, reflection, tool invocation, and knowledge retrieval—before a final answer is generated.

Ada selected OpenAI's models based on three primary technical drivers:

  • Inference Quality: Through a synthetic test framework simulating hundreds of thousands of customers, Ada found that OpenAI models consistently outperformed other major models on their evaluation set.
  • Latency: For real-time voice use cases, Ada reported a significant leap in both quality and latency with the adoption of GPT-4o.
  • Hallucination Control: Ada employs OpenAI's fine-tuning API to generate confidence scores regarding the likelihood of hallucinations in a response, allowing them to minimize inaccuracies within their toolchain.

Impact on Resolution Rates and ROI

The shift to an LLM-based architecture has significantly increased the percentage of queries resolved without human intervention. While the previous version of Ada's product maintained a 70% containment rate with only a 30% resolution rate, the new system maintains similar containment levels but increases the resolution rate to typically 60%, with top-performing customers exceeding 80%.

This increase in resolution rates directly impacts return on investment (ROI) through:

  • Full-time equivalent (FTE) savings.
  • Improved customer satisfaction and retention.
  • Increased new signups.

Future Outlook for AI Automation

Ada's objective is to reach a 100% resolution rate, viewing this goal as a matter of "when, not if." The company is focusing the next 12 months on increasing product transparency and providing more control levers for customers to onboard, measure, and coach AI agents similarly to how they would manage human agents.

Sources