earshot – Giving AI agents a way to listen and make sound

What it is

• A Python‑only, CPU‑only command‑line tool (and library) that turns audio into structured data an AI agent can reason about, and vice‑versa: turn code into sound.
• It ships a set of CLI commands (describe, match, find, critique, …) plus a Python API and an MCP server so Claude‑style agents can call it over a socket.

Key capabilities

Feature	What the agent receives	How it works
describe FILE	A plain‑language description plus calibrated tags (z‑scores), AudioSet labels, loudness, spectral measurements.	Uses two open models: CLAP (text‑audio embeddings) and an AudioSet tagger, plus model‑free measurements.
match FILE QUERY	A similarity score (cosine + z‑score) and a verdict (strong/plausible/weak).	Embeds the query with CLAP, compares to the file’s embedding, and cross‑checks against a vocabulary of 125 reference sounds.
find FILE QUERY	Time stamps of the best matching moments inside a long recording.	Slides a window, scores each segment with the same embedding/tag pipeline, returns top‑k trims.
realness FILE…	Real‑recorded vs code‑synthesised classification with per‑window confidence.	Trained classifier on a held‑out set of real vs synthetic sounds (ROC‑AUC 0.92).
timeline FILE	A “listening transcript”: events with time, level, tag, loudness curve, beat grid, optional video sync.
sheet FILE	An annotated PNG (waveform, spectrogram, tags, sync markers) that visual agents can read.
critique FILE	A scorecard comparing a launch‑style video to a profile built from seven professional launch films.
forge RECIPE	Generate new sounds from code (a “recipe” describing physical synthesis) and iteratively improve them using the same earshot critic.
fit FILE	Analyse a recording and output a Python recipe that can re‑render a close approximation, with knobs for variation.
gallery DIR	Produce a self‑contained HTML page showing many generated sounds, their spectrograms, measurements and recipes.

How agents use it

• The CLI prints a human‑readable summary; --json returns the full structured result, which an LLM‑agent can parse.
• The MCP server keeps the models warm, so Claude Code or other agents can call earshot repeatedly without the 20‑40 s load time.
• Two ready‑made Claude skills (skills/earshot and skills/real-sound) show how to embed the tool in an agent’s workflow.

Making sounds from code – forge

• A recipe is a small Python module that describes physical synthesis blocks (modal resonators, filtered noise, room simulation, etc.).
• earshot forge renders many candidate sounds, scores them with the earshot critic (description match, realness, tonal balance, envelope), and runs an evolutionary search to improve the parameters.
• v0.2.0 adds 34 motion‑design categories (whooshes, clicks, impacts, etc.) with multiple variants and a benchmark that shows the generated sounds are hard to distinguish from real recordings.

Turning recordings into code – fit

• earshot fit performs analysis‑by‑synthesis: it fits a compact synthesis model to the input audio, then writes a reusable recipe.
• Reported median CLAP cosine 0.945 on a 12‑sample test set, and most generated recipes receive the same top tag as the original.

Installation & requirements

• Python 3.10+, ffmpeg/ffprobe on PATH, CPU only.
• First run downloads two open models (~1.4 GB): laion/larger_clap_general and MIT/ast-finetuned-audioset-10-10-0.4593.
• Optional LLM critique uses Gemini (API key supplied by the user).

Performance & limits

• Matching a description is “strong” 98 % of the time; exact variant 61 %.
• Real vs synthetic classification ROC‑AUC 0.92, but the model can be fooled; model‑free measurements (spectral balance, envelope) are more reliable.
• Works on a laptop CPU: analysing a 60‑90 s film takes ~1.5 min.
• Cannot judge high‑level aesthetics, speech content, or frequencies above ~14 kHz.

Who might find it useful?

• Developers building AI‑driven video or game pipelines who need an agent to reason about sound effects.
• Researchers exploring audio‑LLM integration, audio‑to‑code synthesis, or evaluation of synthetic sound.
• Creators who want to generate code‑based sound effects that can be tweaked programmatically.

Bottom line earshot bridges the gap between audio and code, giving AI agents a structured “ear” to listen, evaluate, and generate sound without relying on recorded samples. It is a genuine, open‑source software project focused on AI‑enabled audio analysis and synthesis.