Streaming Whisper in real time

Making a batch speech model transcribe live, on-device, in Rust.

Nov 2025 · Rust · Whisper

Whisper is a batch model: feed it a finished clip and it transcribes it. Turning that into a live tool, where text appears while you're still speaking and matches the final result once you stop, is the actual engineering. It all runs on-device, with Whisper as a pure-Rust model on Candle rather than a whisper.cpp binding.

Why re-running Whisper doesn't work

The naive preview re-runs Whisper on the whole buffer every few hundred milliseconds. It survives one sentence and then falls apart: the same audio gets decoded over and over, and each pass words things slightly differently, so the preview flickers and rewrites text that's already on screen.

The job isn't to decode faster. It's to decode only what changed, and never rewrite what's already on screen.

Segment, then cache the tail

The streaming-Whisper literature solves this at the token level: whisper-streaming's LocalAgreement freezes the longest prefix two successive decodes agree on. The route here is simpler, at the segment level:

Voice-activity detection (VAD, deciding which audio is speech), with hysteresis so it won't flip on every pause, splits the stream into voiced units of at most ten seconds.
Each unit overlaps its neighbour by half a second, so a word on a boundary isn't lost.
Every unit but the last is finished audio, decoded once and reused. Only the final, still-growing unit is live.

That last unit grows every tick, so each unit's text is cached under a key of its start and end sample plus a hash of only the final ~128 ms of audio:

// Cache key: (start_sample, end_sample, tail_hash).
// The hash covers only the last 2048 samples (~128 ms @ 16 kHz),
// so a finished unit's key is stable and the live unit's keeps moving.
const PREVIEW_HASH_TAIL: usize = 2048;

map: HashMap<(usize, usize, u64), String>

A finished unit's tail never changes, so its key is stable and it hits the cache, no Whisper call. Only the live tail keeps moving, so exactly one unit is ever re-decoded.

The live preview and the final transcript run that same path. The final just adds a 120 ms silence pad, so a word spoken as you release the key still lands, and trims the overlapping half-second so no word is doubled at a seam.

The robustness Whisper needs

Left to greedy decoding, Whisper hallucinates: "thanks for watching" over silence, or a looped phrase. The original model's temperature-fallback loop, reimplemented on Candle, keeps it honest:

Decode greedily; if the result fails a compression-ratio check (above 2.4, it's repeating itself) or a log-probability check (below -1.0, low confidence), retry hotter, stepping 0.0, 0.2, up to 1.0.
Drop silence on two signals at once: Whisper's own no_speech_prob above 0.85 and raw audio RMS below 0.003.
Encode once per unit; the features feed both language detection and every decode attempt.

Keeping the audio thread honest

Capture runs on a high-priority OS thread. If its callback blocks even briefly, you get glitches and dropped samples. So it does the minimum and hands each frame to a bounded channel with a non-blocking send:

// Bounded channel; the realtime callback never waits on it.
let (tx, rx) = mpsc::sync_channel::<Vec<f32>>(64);

// Inside the OS audio callback:
let _ = tx.try_send(frame);  // drop on overflow, never block

A worker thread owns the heavy work, resampling to 16 kHz, buffering, triggering decodes, while the realtime thread only produces frames. Push-to-talk is an Arc<AtomicBool> read with a relaxed load, so the hotkey never contends a lock the audio thread touches. Every hop is bounded, so under load the system sheds frames instead of falling behind.

Takeaway

Streaming a batch model is mostly about deciding what not to recompute: segment on silence, commit finished segments, re-decode only the growing edge. Keep the model honest with the fallback loop it was built around, and treat the realtime callback as untouchable.