Home › Whisper model benchmark

Benchmark

Which Whisper model should you run on your Mac?

Picking a local speech model means trading accuracy against speed and download size, and the advice available is mostly guesswork. So we measured it: 6 models, 300 utterances of public benchmark audio on one Apple M1 Max, scored against human transcripts. The short version is that the 4 biggest models are indistinguishable from each other — and the 3.1 GB one is 5.4× the download of the smallest among them for no measurable gain.

Published 7 August 2026

What we measured

Every Whisper model Lesskeys lets you choose, run over the same 300 utterances (37 minutes of audio) from LibriSpeech test-clean — the standard public benchmark set, which ships its own human ground-truth transcripts. We scored word error rate against those transcripts, timed each run, and recorded the download size.

Measured on an Apple M1 Max using whisper.cpp with Metal, GPU and flash attention enabled, whisper.cpp defaults (beam size 5), language forced to en, 8 threads. ● marks the models we treat as tied for best (see below). “89× realtime” means one hour of audio is transcribed in about 1 minute.

The top 4 are a tie, and we are not going to pretend otherwise

Large V3 Q5, Large V3, Large V3 Turbo, Large V3 Turbo Q5 finished within 13 words of each other across 6,086 words of reference text. That is not a meaningful difference — it is the measurement's noise floor. Ranking them would make this page look more decisive and be less true, so we treat any model within 0.5 percentage points of the best as tied.

Which turns the interesting question from “which is most accurate?” into “given they are the same, what does each one cost you?” — and there the gap is enormous.

The 3.1 GB download buys nothing

Of the models tied for best accuracy, the largest is Large V3 at 3.1 GB, running 7× realtime. The smallest is Large V3 Turbo Q5 at 574 MB, running 12× realtime.

Same accuracy, to within 13 words in 6,086. But 5.4× the download and 1.6× slower. If you are choosing a local model on an Apple Silicon Mac and you take one thing from this page, take that: the biggest file is not the best choice, and on this evidence it is not even a defensible one.

What this does not tell you

LibriSpeech test-clean is the easy case. It is read audiobook speech: one person at a time, a good microphone, no crosstalk, no background noise, no accents outside the set. Real meetings and phone calls are materially harder, and every number above would be worse on them. Treat these as a floor and as a fair way to rank the models against each other — not as a promise about your Tuesday standup.

The speed figures come from one machine, an Apple M1 Max. A different Apple Silicon chip will land somewhere else; the ordering should hold, the absolute numbers will not. And this measures the transcription models only — not speaker labelling, which is a separate stage with its own error rate.

Quantisation was effectively free

Both Q5 models are 5-bit quantisations of a full-precision sibling, and in both pairs the smaller file cost nothing you could measure:

• Large V3 Turbo Q5 vs Large V3 Turbo: 574 MB vs 1.6 GB (2.8× smaller), and 4 more errors out of 6,086 words (154 vs 150)
• Large V3 Q5 vs Large V3: 1.1 GB vs 3.1 GB (2.9× smaller), and 2 fewer errors out of 6,086 words (141 vs 143)

This is worth stating carefully: it does not mean quantisation is always free. It means that for these models, on this audio, the penalty was smaller than the noise in the measurement. That is a reason to test it for your own use, not a general law.

Where the differences are real

Below the tied group the gaps stop being noise and start being obvious:

• Small — 3.55%, 75 more errors than the leaders, but 3× the speed at 488 MB
• Tiny — 6.87%, 277 more errors than the leaders, but 8× the speed at 78 MB

Tiny makes 3× as many errors as the leaders — 418 against 141. That is a different kind of transcript, not a slightly worse one. It is still the right choice when you want a rough searchable record in a hurry on a small download, and the wrong one when the transcript is the record.

So which one should you run?

On this evidence: Large V3 Turbo Q5. It is tied for the best accuracy measured here, it is the smallest of that tied group at 574 MB, and at 12× realtime it transcribes an hour of audio in roughly 5 minutes. Going bigger costs download and time and returned nothing measurable; going smaller costs real accuracy.

Reproduce it

The dataset is public (OpenSLR), the models are the standard ggml builds Lesskeys downloads from Hugging Face, and the harness is a single Python file. The sample is chosen deterministically — sorted utterance ids, evenly strided, no random seed — so an independent run picks the identical utterances.

The scoring normaliser is the part worth checking, because a word error rate means nothing without it. Ours ignores differences that are formatting rather than recognition — case, punctuation, hyphenation, 23 versus twenty three, Mr. versus mister — and still counts every real substitution, deletion and insertion. It is locked by eleven test cases, including ones that assert real errors are not normalised away. Without that step models get punished for punctuation style instead of accuracy, which flatters and penalises them roughly at random.

FAQ

Related