BillAI Bass: Building an AI-Powered Voice Assistant with Strands Agents and Amazon Nova 2 Sonic

BillAI Bass is an open-source project that converts a Big Mouth Billy Bass animatronic into a real-time voice assistant. By integrating a Raspberry Pi 5 with Strands Agents' bidirectional streaming (BidiAgent) and Amazon Nova 2 Sonic via Amazon Bedrock, the project enables a physical fish to engage in low-latency voice conversations with synchronized mouth and body movements.

Technical Architecture

The system utilizes a bidirectional streaming architecture to minimize latency and enable natural interaction. Instead of a traditional request-response cycle, audio is streamed live to and from the cloud.

Core Components

  • Brain: Raspberry Pi 5 (8GB recommended) running Raspberry Pi OS Lite (64-bit).
  • AI Model: Amazon Nova 2 Sonic (amazon.nova-2-sonic-v1:0) hosted on Amazon Bedrock in the us-east-1 region.
  • Orchestration: Strands Agents' BidiAgent and BidiAudioIO for managing the bidirectional audio stream.
  • Hardware Control: MX1508 dual H-bridge motor driver to translate Raspberry Pi GPIO signals (3.3V) into the higher current required by the fish's motors.
  • Audio I/O: USB mini speaker and USB gooseneck microphone.

Hardware Implementation

Motor Control and Wiring

The project targets the modern "Gemmy" version of the Big Mouth Billy Bass, which features two motors: one for the mouth and one shared motor for the head and tail.

To safely drive these motors, the MX1508 driver board is used. The wiring connects the Pi's GPIO pins to the driver's inputs (IN1-IN4) and the driver's outputs to the motors. A critical hardware detail is the use of a 27W USB-C PD power supply; standard phone chargers often limit power to USB ports, which can cause instability in the Raspberry Pi 5.

Animatronic Synchronization

Lip-syncing is achieved by measuring the loudness of the audio stream within the playback callback. Because Nova Sonic delivers audio faster than it is played, the system cannot rely on data arrival times. Instead, it analyzes the actual bytes hitting the speaker in real-time:

  • Mouth Movement: Driven by smoothed loudness levels. If the audio exceeds a specific floor (MOUTH_OPEN), the mouth motor is activated.
  • Head Position: The head rises when audio is playing and drops after a period of silence (SILENCE threshold).
  • Tail Flapping: Triggered by loudness peaks (EMPHASIS threshold) or specific conversation events (start/complete/interruption).

Software Configuration

Audio Pipeline

To ensure consistent audio routing, the project uses an .asoundrc configuration file to pin the USB microphone and speaker as the system defaults using card names rather than numbers, as device numbers can shift between reboots.

Security and Identity

For security, the project emphasizes the principle of least privilege. The recommended setup involves a dedicated IAM user with an inline policy that restricts access exclusively to the bedrock:InvokeModelWithBidirectionalStream action for the specific Nova 2 Sonic model ARN.

For production-grade deployments, the project provides an optional AWS IoT Core integration. This replaces long-lived access keys with X.509 certificates, allowing the device to exchange its identity for temporary IAM credentials via the IoT credentials endpoint.

Community Insights and Counterpoints

While the project is praised for its comprehensive documentation and shopping lists, some technical critiques were raised in the community discussion:

"+10 points for giving it context that it's nailed to a plaque. -10 points for using Amazon Bedrock instead of a local conversational model that can run on the Pi without a subscription fee."

Other contributors suggested exploring local models (such as Gemma) to make the device entirely self-contained and remove dependency on cloud subscriptions and internet connectivity.

Troubleshooting and Optimization

Common failure modes identified during the build include:

  • Motor Asymmetry: The mouth motor often fights a return spring and may be stronger in one direction. This is resolved by swapping the two mouth-channel plugs at the Pi header to reverse the drive direction.
  • Audio Echo: The microphone may pick up the speaker's output, causing the AI to interrupt itself. This is mitigated by physical separation of the mic and speaker or lowering the volume.
  • OS Interference: Using the Desktop version of Raspberry Pi OS can lead to the Wayfire panel overwriting the .asoundrc file. The "Lite" version is required for stability.

Sources

Related

  • Dispatch
    OpenAI Low-Latency Voice AI Infrastructure

    OpenAI rearchitected its WebRTC stack using a split relay-plus-transceiver architecture to deliver low-latency, scalable voice AI to over 900 million weekly active users.

  • Dispatch
    Stateful Runtime Environment for Agents in Amazon Bedrock

    OpenAI and Amazon have collaborated to launch a Stateful Runtime Environment in Amazon Bedrock, providing a native AWS infrastructure for AI agents to maintain state, reliability, and governance across multi-step production workflows.

  • Dispatch
    OpenAI gpt-realtime and Realtime API General Availability

    OpenAI has released gpt-realtime, a high-performance speech-to-speech model, and made the Realtime API generally available with support for image inputs, remote MCP servers, and SIP phone calling.

  • Dispatch
    OpenAI Next-Generation Audio Models API Release

    OpenAI has released new speech-to-text and text-to-speech models in its API, featuring improved transcription accuracy and steerable synthetic voices for voice agents.

  • Project
    AISBench/benchmark

    A model evaluation tool based on OpenCompass that provides accuracy and performance benchmarking for both local and service-based AI models across text and multimodal tasks.