BillAI Bass: Building an AI-Powered Voice Assistant with Strands Agents and Amazon Nova 2 Sonic

BillAI Bass: Building an AI-Powered Voice Assistant with Strands Agents and Amazon Nova 2 Sonic

BillAI Bass is an open-source project that converts a Big Mouth Billy Bass animatronic fish into a real-time, voice-interactive AI assistant. By combining a Raspberry Pi 5 with Strands Agents' bidirectional streaming and Amazon Nova 2 Sonic, the project achieves low-latency voice conversations where the fish's mouth lip-syncs to generated audio and its body moves for emphasis.

Technical Architecture

The system relies on a bidirectional streaming pipeline to minimize latency and synchronize physical movement with AI-generated speech.

Core Components

  • Brain: Raspberry Pi 5 (8GB recommended) running Raspberry Pi OS Lite (64-bit).
  • AI Model: Amazon Nova 2 Sonic via Amazon Bedrock, chosen for its high-speed audio generation capabilities.
  • Orchestration: Strands Agents BidiAgent and BidiAudioIO, which handle the bidirectional streaming of audio data between the Pi and the cloud.
  • Hardware Interface: An MX1508 dual H-bridge motor driver allows the Pi's 3.3V GPIO pins to control the 5V motors required for the fish's mouth and body movements.
  • Audio I/O: USB mini-speaker and USB gooseneck microphone.

Lip-Sync and Animation Logic

Because Nova 2 Sonic streams audio faster than it is played back, the project does not trigger animations based on when data arrives. Instead, it implements a custom audio output class (BillyBody) that measures the loudness of the audio samples currently hitting the speaker in the playback callback.

  • Mouth Movement: Driven by smoothed loudness levels; when audio exceeds a specific floor (MOUTH_OPEN), the mouth motor is activated.
  • Head Position: The head rises when audio is actively playing and drops after a period of silence (SILENCE threshold).
  • Tail Flapping: Triggered by loudness peaks (EMPHASIS threshold) and specific conversation events (start, complete, or interruption).

Implementation Guide

AWS Configuration and Security

To prevent security risks associated with storing credentials on an edge device, the project emphasizes a "least privilege" IAM setup. The dedicated billy-bass IAM user is restricted to a single action (bedrock:InvokeModelWithBidirectionalStream) on a single model (amazon.nova-2-sonic-v1:0) in the us-east-1 region.

For production-grade deployments, the project provides an optional AWS IoT Core integration. This replaces long-lived access keys with X.509 certificates, allowing the device to exchange its certificate for temporary IAM credentials via the IoT credentials endpoint.

Hardware Integration

The build involves removing the original control board and replacing it with the MX1508 driver. The wiring map connects the driver to specific Pi GPIO pins:

  • Power: 5V (Pin 4) and Ground (Pin 6).
  • Mouth Control: IN1 (Pin 9) and IN2 (Pin 11).
  • Body Control: IN3 (Pin 15) and IN4 (Pin 13).

Software Setup

The environment requires Python 3.12+ to support Nova Sonic. Key system dependencies include portaudio19-dev for audio access and swig for building the lgpio library. The project uses a virtual environment to manage the strands-agents[bidi,bidi-io] dependency.

Community Insights and Considerations

While the project demonstrates a highly responsive AI toy, community discussions highlight potential areas for optimization:

  • Local LLMs: Some users suggested replacing Amazon Bedrock with local conversational models (such as Gemma) to eliminate subscription fees and cloud dependency.
  • Self-Containment: Suggestions were made to use small, local models that pre-generate responses to make the device entirely self-contained.
  • Hardware Tips: Experienced builders noted the importance of using proper wire strippers rather than diagonal cutters to ensure clean connections.

"Imagine telling that kid [from the 90s] that in the future Billy Bass will have the world's collective knowledge and you'll be able to talk to him about anything."

Troubleshooting Common Issues

Issue Root Cause Solution
IndentationError SSH paste adding leading spaces Use sed -i 's/^ //' filename.py
Weak mouth movement Motor direction asymmetry vs. return spring Swap the two mouth-channel plugs at the Pi header
Audio configuration loss Using Desktop OS instead of Lite Re-flash with OS Lite or disable the Wayfire panel
Invalid sample rate Incorrect ~/.asoundrc config Verify card names using arecord -l and aplay -l

Sources