Whisper: transcription that works

The first recording I fed it was a design review from August, an hour long, with five of us on a video call, two bad microphones and one dog. The transcript got “PgBouncer” right. It got “idempotency key” right. It got everyone’s names right, which the automatic captions in our meeting tool have never once managed. And it kept up with every speaker, more or less, which is the part I’ve been telling people about since.

Our team is spread across a lot of accents, and so are our customers, and every speech recognition system we’ve used has treated most of those accents as an edge case. The captions in our meeting tool do fine with some of us and turn others into nonsense, and the people they mangle have mostly stopped turning them on. On that design review, Whisper did about as well with the colleague the captions have never understood as with anyone else on the call.

What it is

OpenAI released Whisper last Wednesday, the 21st. It’s an encoder-decoder transformer that takes 30-second chunks of audio, as log-Mel spectrograms, and produces text. The interesting part is the training data: 680,000 hours of audio with transcripts collected from the web, about a third of it not in English, which they describe as weakly supervised because those transcripts are messy. OpenAI says plainly that it doesn’t beat the models tuned for LibriSpeech, the standard benchmark, but that across lots of other datasets it makes about half as many errors as those models do, because it hasn’t been tuned to any one of them. That matches what I saw. It’s not perfect on clean audio. It’s very good on real audio.

And they released the code and the weights, under the MIT license, in five sizes from “tiny” (39 million parameters) to “large” (1.55 billion), with English-only versions of all but the largest. That’s what makes it useful to us. You pip install it straight from the GitHub repo, install ffmpeg, and run whisper recording.m4a --model medium.en, and it writes a plain text transcript, plus subtitle files in VTT and SRT with timestamps.

Why local matters

We’ve recorded most of our internal meetings since we went remote, design reviews and incident reviews especially, and almost nobody watches the recordings, because you can’t search an hour of video. Transcripts would fix that. We’ve looked at hosted transcription services twice and said no both times, because those recordings are full of customer names, the occasional screen share of production data, and people being candid about things that went wrong. Sending them to another company means another vendor on our subprocessor list and a conversation with every customer who reads it.

Running a model on our own machines avoids all that. The recordings never leave our account.

On my M1 Air the small English model runs a bit slower than real time on the CPU (it prints a warning that FP16 isn’t supported on CPU and falls back to FP32), so an hour of meeting takes a bit over an hour. That’s fine for trying it out, and not something you’d do to a backlog of two years of recordings. The medium model on the laptop took a lot longer than the meeting had. So on Friday I started a GPU instance in our AWS account, a g4dn.xlarge with one NVIDIA T4, which is about 53 cents an hour on demand, and ran medium.en there. An hour-long meeting took around a quarter of an hour, which works out to a little over ten cents per meeting. I’m sure that can be done faster. I haven’t tried.

What it gets wrong

It doesn’t know who’s talking. There are no speaker labels, just text, so a transcript of five people reads like one person arguing with themselves. For searching that’s fine. For reading, it’s hard work, and we’ll need something else to split the speakers if we want readable minutes.

It also does a strange thing on long silences. Twice, when a recording had a minute or two of nothing (everyone waiting for someone to share their screen), it produced the same sentence over and over, a line that was said before the silence, repeated a dozen times. Once it produced a sentence nobody said at all, something plausible and polite, the kind of filler you might expect in a meeting. That’s the part I’d want people to know about before they trust a transcript as a record. It’s a language model as much as a speech model, and when it doesn’t have audio to go on it fills the gap with likely text.

And punctuation is good but not reliable enough to quote from without checking the recording.

None of that stops us. On Monday I put a little over twenty design and incident reviews through it and dropped the transcripts into the same wiki pages as the recordings, and by the afternoon someone had used search to find the meeting last year where we decided not to split up the billing service and, more usefully, the reason why.