openai/openai-chatkit-advanced-samples

What it solves

This repository provides a collection of advanced, scenario-driven demonstrations of how to build complex AI chat interfaces using the OpenAI ChatKit SDK. It bridges the gap between simple chat bots and fully integrated application experiences by showing how to synchronize AI agents with real-time UI state, external data, and interactive widgets.

How it works

Each example consists of a FastAPI backend using the ChatKit Python SDK and a Vite + React frontend using ChatKit.js. The system enables a bidirectional flow of information:

• Server-to-Client: The backend can stream "client effects" to update the UI (e.g., updating a map or a cat's stats) and send interactive widgets with action configurations.
• Client-to-Server: The frontend can provide the agent with current UI state (e.g., selected map nodes) via client tool calls and send specific commands or entity mentions (@-mentions).
• Agent Integration: Agents use function tools to retrieve application data, manage session state, and trigger UI changes, while specialized "title agents" automatically name conversations.

Who it’s for

Developers building sophisticated AI-powered applications who want to see production-ready patterns for integrating LLMs with interactive frontends, state management, and complex UI components.

Highlights

• Interactive Widgets: Implementation of widgets with both client-side and server-side action handlers for seamless user interaction.
• UI State Sync: Ability for agents to read and mutate the frontend state (e.g., syncing a metro map canvas or customer profiles).
• Context Awareness: Page-aware responses that use request headers to scope AI answers to the current page content.
• Rich Media Support: End-to-end image attachments, speech-to-text dictation, and integrated image generation progress tracking.
• Advanced Navigation: Support for @-mentions of entities (articles, stations) and composer commands (e.g., /care or /find-events) to streamline user input.