spaceamoeba-t/tapq

Multi-modal voice agent for your AI agents. Talk with Claude Code, Codex, and others by voice: answer their prompts, give instructions, ask what they did. Or just nod.

TapQ – Voice‑first supervision for coding agents

What it is – TapQ is a Swift‑based runtime that sits between you and code‑generation agents (Claude Code, Codex, Cursor, OpenCode). It lets the agents ask you for approvals, choices, or follow‑up instructions through your AirPods (or any macOS audio device) so you never have to look at a screen. Your responses are captured via double‑nod / double‑shake gestures, stem‑swipes, or spoken words.

Key capabilities

  • Spoken prompts – When an agent pauses for a permission, a question, or a selection, TapQ reads the prompt aloud in your ear, prefixed with the agent’s name (e.g., “Claude Code: Run swift test. Approve?”).
  • Gesture‑driven answers – A double nod approves, a double shake denies, tilts move between options, and taps confirm selections. The motion data is processed on‑device.
  • Voice interaction – With --voice-backend openai-realtime the spoken reply is sent to OpenAI’s realtime API (only while a response window is open) and can be used for richer commands such as “run the tests and tell me if anything fails” or “when Claude finishes, rerun the tests.”
  • Fallback to screen – If TapQ cannot interpret a gesture or the voice window times out, the original on‑screen prompt is shown unchanged.
  • Local privacy‑first design – Motion and gesture processing stay on the Mac; audio is sent to OpenAI only during a response window, and a local conversation log (wearer‑conversation.jsonl) is bounded to 30 days and can be cleared with tapq memory clear.

How it works

  1. Agent hooks – tapq integration <agent> install injects a small hook or plugin into the target agent. When the agent needs a user decision, the hook forwards the event to the TapQ runtime and waits for a reply.
  2. Runtime – The runtime queues prompts, speaks them through the earbuds, opens a short “response window,” and listens for gestures or voice.
  3. Gesture engine – CoreMotion data from AirPods is interpreted on‑device to detect double nods, double shakes, double tilts, and stem taps/swipes.
  4. Voice backend – Either an on‑device fixed‑vocabulary recognizer (no API key) or OpenAI’s realtime API converts speech into one of the supported actions (approve, deny, select option, queue an instruction, ask a status question, set a follow‑up, start a task).
  5. Result routing – The answer is sent back through the hook to the original agent. If no answer is produced, the hook returns without a response and the agent falls back to its normal UI.

Supported platforms & devices

  • macOS 14+ (Swift 6, Xcode 16 or compatible toolchain) – the full runtime with AirPods integration.
  • Linux – a portable core and CLI that can be built and tested, but without earbuds or agent hooks.
  • AirPods – any model exposing head‑motion (AirPods Pro, AirPods 3+, AirPods Max). Stem‑swipe gestures need AirPods Pro 2 or later.
  • Agents – Claude Code (full hook support), Codex CLI ≥ 0.142.5, Cursor (partial), OpenCode ≥ 1.18.15 (via plugin).

Getting started (requires macOS 14+, Swift 6, and compatible AirPods)

# Clone and build
git clone https://github.com/spaceamoeba-t/tapq.git
cd tapq
swift build && swift test

# Calibrate the motion/voice permissions (runs a headless app so macOS can grant Motion, Speech, Mic)
scripts/run-runtime-app.sh calibration run

# Install hooks for the agents you use (example for Claude Code)
build/TapQRuntime.app/Contents/MacOS/tapq integration claude install --permission-policy native
# …repeat for codex, cursor, opencode as needed

# Run the runtime. The example below enables the OpenAI realtime voice backend and wearer‑gate.
scripts/run-runtime-app.sh serve \
  --voice-backend openai-realtime \
  --voice-instructions --voice-session \
  --wearer-gate --attention wake

When an agent pauses, you’ll hear the prompt in your ear and can respond with a nod, shake, tilt, tap, or spoken command.

Project structure

  • TapQContracts – shared types and protocols used by all adapters.
  • TapQDetectionBaseline, TapQInteractionBaseline, TapQContextBaseline – portable core (gesture detection, state‑machine, memory) that builds on Linux.
  • TapQBrokerRuntime & TapQWireProtocol – local socket broker that mediates between hooks and the runtime.
  • One adapter target per agent (TapQClaudeAdapter, TapQCodexAdapter, etc.) translating hook events.
  • TapQAppleAdapters & TapQVoiceBackends – macOS‑specific motion, speech, and OpenAI realtime integration.
  • TapQCLI – the command‑line interface (tapq and per‑agent hook binaries).

License – Apache 2.0 (source‑only; no Homebrew formula or signed binary yet).

Where it fits – TapQ is not a general‑purpose assistant; it is an interaction layer that lets you stay in the physical world (earbuds, head gestures) while supervising multiple coding agents that would otherwise require you to stare at a screen.

Related

  • Project
    Stability-AI/stable-audio-metrics

    A collection of evaluation metrics for music and audio generative models, providing tools to calculate Fréchet Distance, KL divergence, and CLAP scores for long-form stereo audio.

  • Project
    Prorise-cool/Claude-Code-Multi-Agent

    Claude Code Multi‑Agent is an open‑source framework that gives Anthropic’s Claude Code project awareness and expert‑agent capabilities. It uses a Python Hook system and a locally‑run Ollama model to automatically detect the codebase type, analyse user intent, load 300+ domain‑specific “Skills” (backend, frontend, testing, security, etc.), and enforce automatic documentation (DEVELOPMENT.md, KNOWLEDGE.md, CHANGELOG.md). After cloning, installing Ollama, a model, and uv, you drop your project into the repo and Claude Code instantly knows its context, suggests appropriate tools, and can be driven with slash commands like `/backend‑specialist` or workflow commands such as `/kiro/spec`. Key points: - Zero‑config start; all configuration lives in `.claude/settings.json` and `prompts.json`. - Hooks run on session start, user prompt, tool use, etc., handling project detection, intent analysis, and doc updates. - Skills are defined by `SKILL.md` files and are auto‑discovered. - Works entirely locally (no external API), integrates with Git, and is MIT‑licensed. Ideal for developers or small teams who want a reproducible, on‑premise AI coding assistant.

  • Project
    Intelligent-Internet/zenith

    Zenith is a Python harness that lets Claude Code, Codex, or Hermes run as long‑running coding agents. An orchestrator repeatedly checks project state, spawns worker/tester sub‑agents, registers reusable skills, replans, and only stops when the task is truly complete. Benchmarked on the Frontier SWE suite, Zenith (using GPT‑5.5) achieves the best average rank while costing less than half of the strongest baseline (RALPH). The repo provides a CLI, MCP server, ACP adapters, and a full technical report.

  • Project
    gdstudio-org/Embeat

    A music recommendation system that combines acoustic feature encoding via MLP and collaborative filtering via Track2Vec to provide high-quality, genre-aware song suggestions.