For a decade, production ASR was a pipeline: acoustic model, pronunciation dictionary, language model, often with a domain-specific fine-tune. Whisper treats transcription as sequence-to-sequence on log-mel spectrograms, with special tokens that request language, task (transcribe vs translate), and timestamps. The training set is 680,000 hours of audio paired with transcripts harvested from the web, filtered rather than hand-curated in a studio.

The result that mattered for engineers was robustness. Models trained on clean read speech fall over on YouTube, accents, and background noise. Whisper's bet is that the mess is the training distribution. Zero-shot English ASR on several out-of-distribution sets approached or beat fine-tuned specialist systems, and multilingual transcription plus X-to-English translation came from the same checkpoint. That is why so many products started from an off-the-shelf Whisper weight instead of a Kaldi recipe.

Multitask tokens are the control plane

The decoder is a language model over a byte-level BPE vocabulary plus task tokens. You can ask for timestamps, or translation, or a language ID. Those tokens are not a product afterthought; they are how one set of weights serves many endpoints. If you strip them and decode like a vanilla captioner, you give up the paper's interface.

The encoder is a convolutional stem plus transformer over audio frames. Chunking long files (the 30-second windows in the original inference recipe) is a production detail you will re-learn the first time a two-hour meeting overflows memory or loses context at boundaries. Overlap-and-stitch is a systems problem the paper's eval clips do not fully simulate.

Weak labels, strong filters

Web transcripts are wrong, misaligned, and sometimes machine-generated. The authors filter, which is the actual craft. If you reproduce "train on all the audio we can download" without the filters, you will train a model that transcribes ads, repeats, and hallucinates YouTube title case. Whisper's hallucinations on silence and on non-speech are a known operational issue: the decoder is a language model and will talk if you let it.

Language identification and code-switching remain sharp edges. A meeting that flips between languages can get a single language token for a chunk and then a bad transcript. Do not treat the paper's multilingual averages as a promise for your bilingual call center without measuring that mix.

A worked meeting pipeline

You VAD-split a call, run Whisper per chunk with transcribe and English forced when your product is English-only, then merge timestamps. You add a rejection rule: if the audio energy is near silence and the decoder emits a full sentence, drop it. You compare WER on a private 50-hour set, not on LibriSpeech test-clean. The paper already told you that in-distribution read speech is the easy number.

Failure modes

Hallucinated captions on music, silence, or logos. Gate with energy or a speech classifier.

Timestamp drift after stitching 30-second windows. Overlap and constrain.

Fine-tuning on 20 hours of in-domain audio and destroying multilingual robustness you needed next quarter.

Evaluating only WER when your product needs speaker attribution or PII redaction. Whisper does not give you diarization.

What you can borrow

  • Prefer large, noisy, weakly labeled supervision over a small clean set when the deployment audio is the internet.
  • Put task and language control in tokens so one model serves transcribe, translate, and timestamps.
  • Filter web labels; scale without filters is garbage-in.
  • Add silence and hallucination guards in the serving path.
  • Do not use vanilla Whisper as a speaker-attributed, personally-identifiable-information-safe archive without extra systems.