Automatic speech recognition used to mean a carefully licensed corpus and a model that broke on accents, noise, and languages it had never been funded to collect. OpenAI's Whisper trained an encoder-decoder Transformer on hundreds of thousands of hours of weakly labeled web audio — transcripts of varying quality, many languages, messy conditions — and showed that scale plus diversity beat a lot of specialist architecture for robustness. The 2022 paper, and the open weights that followed, made "good enough transcription" a library import. Production stacks then discovered that a research forward-pass is not a product: files are hours long, users want streaming captions, timestamps drift, and hallucination on silence is a support ticket.
Weak supervision is a data pipeline
Whisper's bet was that imperfect transcripts, in bulk, teach more than a small gold set. Filtering still mattered: the paper describes quality heuristics so the model is not trained to imitate garbage. Multilingual and multitask training (transcribe, translate to English, language ID) shares an encoder, which is why one checkpoint covers so many locales. That sharing is also why a rare language can be pulled around by English-heavy data. Product teams serving a single locale sometimes fine-tune or post-process rather than trusting zero-shot equally everywhere.
The open weights created a fork in serving: call OpenAI's API, or run whisper (and later faster C++/TensorRT ports) on your GPUs. APIs hide batching and model choice (tiny through large-v2 and successors). Self-hosting hides nothing: VRAM, batch size, and the 30-second spectrogram window the original model was built around.
Chunking, timestamps, and the silence hallucination
Long-form audio is sliced. Naive slicing cuts words; overlapping windows and VAD (voice activity detection) are the usual repair. Whisper's timestamp tokens help alignment but can fail on music, overlapping speakers, and long pauses. A known failure is emitting plausible sentences when the input is silence or hum — the decoder language model prior winning over acoustics. Production systems add energy gates, repetition detection, and sometimes a second pass. Word-error-rate on LibriSpeech will not catch that.
Latency requirements split the design. Offline meeting notes can use large models and beam search. Live captions need streaming or small models, partial hypotheses, and a UI that can revise. Punctuation and inverse text normalization (numbers, dates) are often a separate layer; raw Whisper output is not what a legal transcript should look like.
The borrow from Whisper is the training philosophy and the serving checklist: VAD, chunk overlap, hallucination guards, language hints when you know the locale, and a model size chosen against a latency budget rather than a leaderboard.
Speaker overlap is still a gap. Whisper is not a full diarization stack; meeting products usually chain a separate speaker model and then align words to turns. If you skip that, transcripts look fluent and attribute speech to the wrong person, which is worse than a slightly higher word-error rate. Budget diarization as its own SLO when the product is minutes, not voicemail.
What you can borrow
- Treat noisy, diverse audio as a feature of the training set; still filter the worst labels.
- Never run hour-long files as one decode; chunk with overlap and a speech detector.
- Guard against silence hallucinations with energy/VAD and repetition checks, not WER alone.
- Pick model size from latency and GPU memory;
largeis not the default for live captions. - Pass a language hint when the product knows it; multilingual encoders still skew toward high-resource languages.