NVIDIA Nemotron Voice Agent and Open Models for Interactive Characters

Overview of MonsterHouse

MonsterHouse is an interactive, voice-responsive installation that uses a suite of NVIDIA AI tools to create a character capable of listening to, conversing with, and reacting to visitors. The project demonstrates the capabilities of the NVIDIA Nemotron Voice Agent, which integrates speech recognition, small language models (SLMs), text-to-speech (TTS), and 3D facial animation to create a self-contained, low-latency conversational agent.

The Technical Stack

The system coordinates multiple specialized models to handle the full lifecycle of an interaction, from perception to animation:

  • Listening: NVIDIA Nemotron Speech ASR (streaming) converts visitor speech into text.
  • Thinking: NVIDIA Nemotron 3.5 Bolt E4B 4GB, an open small language model, generates responses. A custom LoRA adapter provides the character's personality.
  • Speaking: NVIDIA Magpie TTS Zeroshot generates the voice from a short reference sample, allowing for distinct emotional delivery.
  • Animating: NVIDIA Audio2Face-3D translates the generated speech into facial animations within Unreal Engine 5.
  • Seeing: NVIDIA Nemotron 3 Nano Omni provides visual context, while YOLO and ByteTrack handle visitor counting and tracking.

Performance and Optimization

To ensure a responsive experience, the system utilizes a hybrid approach to response generation. By caching common lines and pre-synthesizing audio, the system achieves a median end-of-speech-to-audio latency of 0.72 seconds for cached responses, compared to 1.70 seconds for live generative responses. The pipeline is distributed across two GPUs—an NVIDIA RTX PRO 6000 Blackwell Max-Q for the agent logic and an NVIDIA RTX PRO 4000 Blackwell for Unreal Engine rendering—to maintain consistent frame rates and audio-visual synchronization.

Implementation Strategy

For developers looking to build similar interactive characters, the following workflow is recommended:

  1. Pipeline Foundation: Start with the NVIDIA Nemotron Voice Agent to handle the core logic of speech activity detection and utterance handling.
  2. Model Selection: Utilize Bolt E4B as the primary brain, as it is purpose-built for local conversational agents and offers a smaller footprint than larger models.
  3. Personality Tuning: Use system prompts and example dialogues for basic behavior. For more complex character traits, train a LoRA adapter using the NVIDIA NeMo AutoModel recipe.
  4. Voice and Animation: Apply Magpie TTS Zeroshot for zero-shot voice conditioning and use Audio2Face-3D to drive 3D character rigs. Clean speech should be used for animation, with vocal effects applied as a separate audio stream.
  5. Visual Context: Integrate vision-language models like Nemotron 3 Nano Omni if the character requires awareness of its environment or specific visual inputs.

Sources