How Tolan Builds Voice-First AI with GPT-5.1

Tolan, developed by Portola, is a voice-first AI companion designed for open-ended, long-term dialogue. By leveraging GPT-5.1 and a specialized architecture for context and memory, Tolan achieves the low latency and personality consistency required for natural, meandering conversations.

GPT-5.1 Enhances Steerability and Latency

GPT-5.1 provides the critical technical gains in steerability and latency that enable Tolan's character-driven experience. The transition to GPT-5.1, combined with the Responses API, reduced speech initiation time by over 0.7 seconds, significantly improving conversational flow.

Beyond speed, GPT-5.1 improved the model's ability to follow complex, layered prompt instructions—including tone scaffolds, memory injections, and character traits—with far less drift over long interactions. According to Quinten Farmer, CEO of Portola, "GPT-5.1 gave us the steerability to finally express the characters we had in mind."

Real-Time Context Reconstruction

To handle the volatility of voice conversations where users frequently shift topics midstream, Tolan avoids caching prompts across multiple turns. Instead, the system rebuilds its context window from scratch every turn. Each reconstruction incorporates:

  • A summary of recent messages
  • A persona card
  • Vector-retrieved memories
  • Tone guidance
  • Real-time app signals

This approach allows the AI to adapt instantly to abrupt topic changes, ensuring the agent remains grounded without relying on large, fragile prompts.

Memory Architecture and Personality Management

Tolan employs a modular memory system to maintain coherence over time, storing not only facts but also emotional "vibe" signals.

Technical Implementation of Memory

  • Embedding and Storage: Memories are embedded using the text-embedding-3-large model and stored in Turbopuffer, a high-speed vector database enabling sub-50ms lookup times.
  • Retrieval: Memory recall is triggered by the user's latest message and system-synthesized questions.
  • Maintenance: A nightly compression job removes redundant or low-value entries and resolves contradictions to maintain memory quality.

Personality and Emotional Tuning

Each AI character is based on a character scaffold authored by a science fiction writer and refined by a behavioral researcher. A parallel system monitors the emotional tenor of the conversation, dynamically adjusting the delivery to shift between playful or grounded tones based on user cues while maintaining the core personality.

Performance Metrics and User Impact

Since its February 2025 launch, Tolan has reached over 200,000 monthly active users and a 4.8-star App Store rating. The implementation of GPT-5.1-powered personas led to measurable improvements in user experience:

  • Memory recall misses: Decreased by 30% (measured via in-product frustration signals).
  • Next-day user retention: Increased by more than 20%.

Core Principles for Voice AI Development

Based on their development process, Portola identifies four key principles for building natural voice agents:

  1. Design for conversational volatility: Systems must pivot quickly as voice conversations shift mid-sentence.
  2. Treat latency as part of the product experience: Sub-second responsiveness is essential to avoid a mechanical feel.
  3. Build memory as a retrieval system: High-quality compression and fast vector search are more effective than oversized context windows.
  4. Rebuild context every turn: Regenerating context prevents drift and keeps agents grounded during meandering conversations.

Sources