QwenAudio/qwen-audio-agent

Qwen Audio Agent – Real‑time Voice‑first AI Assistant

What it is – A Node‑JS based runtime that lets you talk to an AI agent (e.g., Qwen 3.5‑Omni, Qwen Audio) in full‑duplex, low‑latency voice. While you keep the conversation flowing, the agent can launch background tasks (code generation, file edits, web searches, etc.) and report progress back into the same voice chat. The system works as a command‑line gateway, a web UI, a terminal TUI, or a floating desktop “orb” that lives on macOS, Windows and Linux.

Key capabilities

• Full‑duplex realtime voice – Speak, interrupt, and hear the assistant without waiting for a turn‑based response.
• Realtime model catalog – Choose DashScope Qwen Audio or Qwen 3.5‑Omni realtime models from a shared catalog.
• Backend Agent integration – Plug in a coding‑oriented agent (Qwen Code, OpenCode, Kimi Code, DeepSeek, etc.) via the ACP architecture; the frontend can also run in a frontend‑only mode with no backend.
• Parallel task handling – Launch multiple independent tasks, watch live progress cards, ask for status, or cancel at any time. Results are automatically spoken back to you.
• Cross‑platform UI – Web UI, terminal TUI, and a desktop floating orb with custom skins and pet animations.
• Personalization & memory – Per‑user long‑term preferences and a local memory file that persists across sessions.
• Extensible gateway – One‑click installation of “Skills”, tool adapters, and custom voice front‑ends (including a fully‑local speech‑to‑speech stack).

How to get started

1. Install globally with npm (requires Node 22.22.2+):

   npm install -g qwen-audio-agent
   

2. Run the config wizard to store your DashScope API key and choose a realtime model:

   qwenaudio config
   

3. Start the gateway (the local server that mediates voice and the backend agent) and a UI:

   qwenaudio          # starts the gateway
   qwenaudio tui      # opens the terminal UI (or `qwenaudio webui` for the browser)
   

   On macOS/Windows/Linux you can also launch the desktop orb with npm run desktop or download a pre‑built installer from the releases page.

Typical use cases

• Desktop productivity – Talk to the assistant while it writes code, edits files, or runs long builds; ask “how far along is the compilation?” and get spoken updates.
• Smart cockpit – Voice control of vehicle navigation, music, weather, and quick‑buy flows (provided as an example in examples/car).
• Customer‑support bots – Keep a natural conversation while the agent looks up orders or creates tickets in the background.
• Embodied agents & livestream assistants – Combine voice chat with physical actions or live‑stream interactions, with the ability to pause, resume, or cancel tasks.

Architecture at a glance

• Frontend (voice) – Captures microphone audio, runs VAD, sends audio/text to a realtime model (DashScope or local speech‑to‑speech).
• Gateway – Central Node process that routes messages, manages task lifecycles, and maintains per‑user memory.
• Backend Agent – Optional ACP‑compatible agent that performs reasoning, tool use, or code generation. The gateway can run multiple agents in parallel.

Extensibility

• Add new Skills (tool adapters) via the built‑in skill manager.
• Swap the voice frontend to a fully local speech‑to‑speech stack (no cloud key needed).
• Create custom desktop skins or pet animations that react to runtime states.

Community & support

• GitHub issues for bugs and feature requests.
• WeChat groups (QR codes in the repo) for Chinese‑language community chat.
• Detailed docs: user guide, quick‑start, architecture deep‑dive, and privacy notice.

License – Apache 2.0.

---

All information above is taken directly from the repository’s README; no external assumptions have been added.