Audio Generation from Text and Reference Sound: Current Landscape and Available Models

No Fully Commercial Text‑plus‑Reference‑Audio Model Exists Yet

There is currently no widely‑available, production‑grade model that accepts a reference sound and a textual instruction and outputs a new, matching audio clip. Existing commercial services such as ElevenLabs focus on pure text‑to‑speech, while research prototypes either produce poor quality or require complex workarounds.

Voice‑Cloning Systems Offer a Partial Solution

Voice‑cloning models can ingest a short audio sample and a text prompt to generate speech that mimics the reference voice. For example, Qwen‑3 TTS includes a “clone” feature that reads arbitrary text using the timbre of a supplied voice sample. Users have reported satisfactory results for personalized audiobooks, but the output is limited to spoken language rather than arbitrary sound effects.

"I used QWEN3 TTS clone to make a customised audiobook for my mother. It worked quite well." – @jajazheng

Research Prototypes Target Audio Generation

Several open‑source projects aim to generate audio conditioned on text and/or reference embeddings:

• AudioLDM / AudioLDM‑2 and AudioX are diffusion‑based models that can synthesize audio from textual descriptions. They are still experimental and often produce low‑fidelity results, especially for non‑musical sound effects.

  "Apparently the AudioX and AudioLDM(2) models do this but I think you've found a genuine gap." – @chr15m

• Stable Audio 3 claims to support text‑plus‑reference generation, but early users report terrible audio quality, indicating the technology is not yet production‑ready.

• Suno/Lyria are music‑generation models that could be repurposed for sound‑effect synthesis by isolating the desired elements after generation, though they are primarily trained on musical data.

  "Even though they're technically trained for music, it might be worth testing Suno/Lyria to see if they're able to do this." – @moonu

Embedding‑Based Retrieval and Conditioning

Projects like CLAP (Contrastive Language‑Audio Pre‑training) generate audio embeddings analogous to CLIP for images. While CLAP does not directly synthesize audio, its embeddings can be used to retrieve or rank existing sound libraries based on textual queries, serving as a potential component in a larger pipeline.

"CLAP generates audio embeddings – much like CLIP does for visual inputs." – @bobosha

Indirect Workflows Using Spectrogram Manipulation

A creative workaround is to treat audio as an image: convert the reference audio to a spectrogram, feed the spectrogram and text prompt to an image generation model (e.g., Stable Diffusion), then invert the modified spectrogram back to waveform. This approach is untested at scale and may introduce artifacts, but it demonstrates how existing image models can be repurposed for audio tasks.

"How good are image models with manipulating spectrogram images? Convert input audio to spectrogram → pass spectrogram + prompt to an image model → convert back to audio waveform." – @xg15

Code‑Generation for Procedural Synthesis

Another angle is to generate code that synthesizes sounds (e.g., using SuperCollider or Faust). While not a direct text‑plus‑reference model, AI‑generated code can produce custom sound effects when combined with domain‑specific knowledge.

"You should look into using AI to generate code that synthesizes sounds." – @soundworlds

Emerging Platforms and Services

• Daydream Music offers a realtime engine with integrations for audio tools; however, it focuses on music creation rather than arbitrary sound‑effect generation.

  "Daydream Music – the DEMON realtime engine is open source. The hosted service has a number of integrations with popular audio tools." – @jallmann

Summary of the Landscape

• No turnkey commercial product currently fulfills the exact text + reference‑audio → new audio use case.
• Voice‑cloning tools (e.g., Qwen‑3 TTS) can mimic a reference voice but are limited to speech.
• Research diffusion models (AudioLDM, AudioX) provide experimental capabilities with limited fidelity.
• Embedding models (CLAP) enable retrieval but not synthesis.
• Spectrogram‑image pipelines and code‑generation are promising research directions but require substantial engineering.

Bottom line: The field is still in an early stage comparable to image generation circa 2016. Practitioners must combine existing voice‑cloning, diffusion prototypes, or creative workarounds to approximate the desired functionality until a mature, commercial solution emerges.