RealtimeVAD 0.1.0: Early Voice‑Activity Detection Gives Voice Assistants Up to 130 ms Faster Turn Completion
Quick takeaway
RealtimeVAD provides a speculative pause event that fires median 92 ms earlier than Silero VAD (up to 130 ms in fast mode), enabling voice‑assistant pipelines to start transcription, LLM calls, or TTS earlier and cancel them if the speaker continues, while preserving Silero’s final end accuracy.
What RealtimeVAD does
- Hybrid detection – RealtimeVAD runs Silero VAD to confirm speech, but every 10 ms it also compares the current audio level to the room‑noise floor and the speaker’s own level.
- Early
pauseevent – When the audio drops below the adaptive floor, apauseevent is emitted before Silero’s probability falls below 0.5. - Speculative pipeline – Applications can start costly work at
pauseand abort it on a subsequentresume(≈20 ms after sound returns) or finalize at the guaranteedendevent. - Drop‑in compatibility – The API mirrors Silero’s
VADIterator, and a LiveKit plugin is provided for agents.
Measured lead over Silero VAD
| Condition | Median lead (ms) | % of turns where pause is earlier |
|---|---|---|
| Close‑mic, noise‑suppressed, single speaker (default) | 92 | 92 % |
Same hardware, pause_mode="fast" |
130 | 99 % |
| Close‑mic, 214 speakers (no noise suppression) | 82‑90 | 80‑90 % |
| Quiet speech, –38 dBFS | 100‑122 | 91‑95 % |
| Simulated phone line | 92‑114 | 79‑87 % |
| Distant table mic (AMI) | 0 | 31 % |
| Loud steady noise (≈speaker level) | none | – |
| Babble/music close to speaker | expect none | – |
The end event is identical to Silero’s end; no turn is cut earlier than Silero would cut it.
How pause works in practice
"
pausefires on every short gap between words, about twelve times as often as Silero’s own speech‑off. Most pauses are followed by more speech. The gain comes from starting the transcript, the LLM call or the voice output atpauseand throwing that work away when the speaker continues (speculative generation)." – RealtimeVAD README
- Start work you can cancel – At
pause, launch transcription, LLM inference, or TTS. The work must be cheap to abort (e.g., streaming LLM, pre‑emptive generation). - Cancel on
resume–resumearrives ~20 ms after sound returns, reclaiming ~90‑95 % of premature pauses. - Never treat
pauseas final – Onlyendmarks the true turn boundary. - Fallback – In 1‑4 % of clips the last
pauseis withdrawn and no newpauseappears beforeend; the system should fall back toend.
Choosing the right pause_mode
| Mode | Median lead (ms) | % of pauses >100 ms early | % >200 ms early | Avg. premature pauses per second |
|---|---|---|---|---|
"accurate" (default) |
82‑90 (92 for 1‑speaker set) | 1‑9 % | 0.1‑3 % | 1.7‑2.2 |
"balanced" |
102‑114 | 11‑17 % | 2‑7 % | 1.7‑2.0 |
"fast" |
118‑126 (130 for 1‑speaker set) | 18‑24 % | 4‑10 % | 2.0‑2.2 |
When to use fast – If downstream processing tolerates missing the last word fragment (e.g., it will be re‑generated at end). Otherwise stick with the default to keep early pauses more accurate.
Performance and resource footprint
- CPU usage – ~8 ms of processing per second of audio on a single thread.
- Dependencies – Only
numpyandonnxruntime; the Silero model is bundled, no PyTorch required. - Audio format – 16 kHz mono (float32 or int16). For 8 kHz audio you must fall back to pure Silero.
Integration example (Python)
from RealtimeVAD import VAD
vad = VAD(resume_events=True) # enable `resume`
for chunk in microphone_chunks(): # 16 kHz mono, any chunk size
for ev in vad.feed(chunk):
if ev.kind == "start":
stream_to_asr(vad.audio())
elif ev.kind == "pause":
speculate(vad.audio()) # start LLM, TTS, etc.
elif ev.kind == "resume":
cancel_speculation()
elif ev.kind == "end":
finalize(ev.segment.audio) # turn is truly over
The vad.audio() method always returns the running segment, including ~0.3 s of pre‑speech audio.
Real‑world impact (LiveKit case study)
- Using the LiveKit Agents plugin, the first assistant audio arrived 120 ms sooner than with Silero alone when Whisper was the recognizer, 56 ms sooner with a streaming recognizer, and 0 ms improvement when the LLM itself was already faster than the turn.
- The trade‑off: speculative requests increased up to 1.7× in pipelines that could not cancel quickly, highlighting the importance of fast‑cancellable downstream components.
Limitations
- Speculative only – If your pipeline cannot cancel work,
pauseadds noise without benefit. - No lead on distant mics or loud, near‑speaker noise – In those scenarios RealtimeVAD behaves exactly like Silero.
- Early pauses are not always correct – 1‑2 % of read sentences and 6‑9 % of meeting clips have a last
pause>100 ms before the word ends. - Silero’s inherent limits remain – Whisper‑style speech, laughter, coughs, and very quiet speech may still be missed.
- Alpha‑stage API – Future releases may change constructor arguments or defaults.
Community discussion highlights
"You can also avoid the problem altogether by using a speculative regeneration pipeline that triggers on a fixed silence window (e.g., 300 ms). In my setup the assistant starts answering within 0.9‑1.2 s after the user stops speaking." – nowittyusername (HN comment)
"Run a tiny LLM that predicts the probability the user is done speaking and use that to shorten VAD time. It isn’t as good as a true speech‑to‑speech model but cuts a few hundred ms." – com2kid (HN comment)
These comments reinforce that RealtimeVAD’s value lies in speculative pipelines and that alternative approaches (fixed silence windows or predictive models) aim at the same latency reduction.
When to adopt RealtimeVAD
- Your pipeline can start work early and cancel it within tens of milliseconds (e.g., streaming ASR → LLM → TTS with pre‑emptive generation).
- You have a close‑mic capture chain (desktop or mobile with noise suppression). Distant microphones will see no benefit.
- You accept occasional premature pauses and have logic to handle
resumeor fallback toend.
If any of these conditions are false, the simpler Silero VAD remains the better choice.
How to switch from Silero to RealtimeVAD
# Old Silero import
from silero_vad import load_silero_vad, VADIterator
# New RealtimeVAD import (drop‑in)
from RealtimeVAD import VAD
vad = VAD() # default settings give the 92 ms median lead
The returned events (start, pause, resume, end) match Silero’s VADIterator plus the new speculative signals.
License
RealtimeVAD is released under the MIT license. It bundles the MIT‑licensed Silero VAD model.