RealtimeVAD 0.1.0: Early Voice‑Activity Detection Gives Voice Assistants Up to 130 ms Faster Turn Completion

Quick takeaway

RealtimeVAD provides a speculative pause event that fires median 92 ms earlier than Silero VAD (up to 130 ms in fast mode), enabling voice‑assistant pipelines to start transcription, LLM calls, or TTS earlier and cancel them if the speaker continues, while preserving Silero’s final end accuracy.


What RealtimeVAD does

  • Hybrid detection – RealtimeVAD runs Silero VAD to confirm speech, but every 10 ms it also compares the current audio level to the room‑noise floor and the speaker’s own level.
  • Early pause event – When the audio drops below the adaptive floor, a pause event is emitted before Silero’s probability falls below 0.5.
  • Speculative pipeline – Applications can start costly work at pause and abort it on a subsequent resume (≈20 ms after sound returns) or finalize at the guaranteed end event.
  • Drop‑in compatibility – The API mirrors Silero’s VADIterator, and a LiveKit plugin is provided for agents.

Measured lead over Silero VAD

Condition Median lead (ms) % of turns where pause is earlier
Close‑mic, noise‑suppressed, single speaker (default) 92 92 %
Same hardware, pause_mode="fast" 130 99 %
Close‑mic, 214 speakers (no noise suppression) 82‑90 80‑90 %
Quiet speech, –38 dBFS 100‑122 91‑95 %
Simulated phone line 92‑114 79‑87 %
Distant table mic (AMI) 0 31 %
Loud steady noise (≈speaker level) none –
Babble/music close to speaker expect none –

The end event is identical to Silero’s end; no turn is cut earlier than Silero would cut it.


How pause works in practice

"pause fires on every short gap between words, about twelve times as often as Silero’s own speech‑off. Most pauses are followed by more speech. The gain comes from starting the transcript, the LLM call or the voice output at pause and throwing that work away when the speaker continues (speculative generation)." – RealtimeVAD README

  • Start work you can cancel – At pause, launch transcription, LLM inference, or TTS. The work must be cheap to abort (e.g., streaming LLM, pre‑emptive generation).
  • Cancel on resume – resume arrives ~20 ms after sound returns, reclaiming ~90‑95 % of premature pauses.
  • Never treat pause as final – Only end marks the true turn boundary.
  • Fallback – In 1‑4 % of clips the last pause is withdrawn and no new pause appears before end; the system should fall back to end.

Choosing the right pause_mode

Mode Median lead (ms) % of pauses >100 ms early % >200 ms early Avg. premature pauses per second
"accurate" (default) 82‑90 (92 for 1‑speaker set) 1‑9 % 0.1‑3 % 1.7‑2.2
"balanced" 102‑114 11‑17 % 2‑7 % 1.7‑2.0
"fast" 118‑126 (130 for 1‑speaker set) 18‑24 % 4‑10 % 2.0‑2.2

When to use fast – If downstream processing tolerates missing the last word fragment (e.g., it will be re‑generated at end). Otherwise stick with the default to keep early pauses more accurate.


Performance and resource footprint

  • CPU usage – ~8 ms of processing per second of audio on a single thread.
  • Dependencies – Only numpy and onnxruntime; the Silero model is bundled, no PyTorch required.
  • Audio format – 16 kHz mono (float32 or int16). For 8 kHz audio you must fall back to pure Silero.

Integration example (Python)

from RealtimeVAD import VAD
vad = VAD(resume_events=True)  # enable `resume`

for chunk in microphone_chunks():   # 16 kHz mono, any chunk size
    for ev in vad.feed(chunk):
        if ev.kind == "start":
            stream_to_asr(vad.audio())
        elif ev.kind == "pause":
            speculate(vad.audio())          # start LLM, TTS, etc.
        elif ev.kind == "resume":
            cancel_speculation()
        elif ev.kind == "end":
            finalize(ev.segment.audio)       # turn is truly over

The vad.audio() method always returns the running segment, including ~0.3 s of pre‑speech audio.


Real‑world impact (LiveKit case study)

  • Using the LiveKit Agents plugin, the first assistant audio arrived 120 ms sooner than with Silero alone when Whisper was the recognizer, 56 ms sooner with a streaming recognizer, and 0 ms improvement when the LLM itself was already faster than the turn.
  • The trade‑off: speculative requests increased up to 1.7× in pipelines that could not cancel quickly, highlighting the importance of fast‑cancellable downstream components.

Limitations

  • Speculative only – If your pipeline cannot cancel work, pause adds noise without benefit.
  • No lead on distant mics or loud, near‑speaker noise – In those scenarios RealtimeVAD behaves exactly like Silero.
  • Early pauses are not always correct – 1‑2 % of read sentences and 6‑9 % of meeting clips have a last pause >100 ms before the word ends.
  • Silero’s inherent limits remain – Whisper‑style speech, laughter, coughs, and very quiet speech may still be missed.
  • Alpha‑stage API – Future releases may change constructor arguments or defaults.

Community discussion highlights

"You can also avoid the problem altogether by using a speculative regeneration pipeline that triggers on a fixed silence window (e.g., 300 ms). In my setup the assistant starts answering within 0.9‑1.2 s after the user stops speaking." – nowittyusername (HN comment)

"Run a tiny LLM that predicts the probability the user is done speaking and use that to shorten VAD time. It isn’t as good as a true speech‑to‑speech model but cuts a few hundred ms." – com2kid (HN comment)

These comments reinforce that RealtimeVAD’s value lies in speculative pipelines and that alternative approaches (fixed silence windows or predictive models) aim at the same latency reduction.


When to adopt RealtimeVAD

  • Your pipeline can start work early and cancel it within tens of milliseconds (e.g., streaming ASR → LLM → TTS with pre‑emptive generation).
  • You have a close‑mic capture chain (desktop or mobile with noise suppression). Distant microphones will see no benefit.
  • You accept occasional premature pauses and have logic to handle resume or fallback to end.

If any of these conditions are false, the simpler Silero VAD remains the better choice.


How to switch from Silero to RealtimeVAD

# Old Silero import
from silero_vad import load_silero_vad, VADIterator

# New RealtimeVAD import (drop‑in)
from RealtimeVAD import VAD

vad = VAD()  # default settings give the 92 ms median lead

The returned events (start, pause, resume, end) match Silero’s VADIterator plus the new speculative signals.


License

RealtimeVAD is released under the MIT license. It bundles the MIT‑licensed Silero VAD model.

Sources