Home › Silent truncation

Measurement

When Whisper stops early and says it finished

We ran 5 Whisper models across 29 languages through the shipping app. 10 runs came back with part of the audio transcribed and no error raised. Here is the data, the detection method, and what it says about the models we offer.

Published 24 August 2026

The failure mode nobody scores for

A speech model can fail in two directions. It can produce the wrong words — that is what word error rate measures, and it is what every benchmark reports. Or it can simply stop: return a transcript that is correct as far as it goes, covering the first fraction of the audio, and hand it back as though it were finished.

The second failure is much worse in practice and much rarer in benchmarks, because a transcript that stops early still scores as "a result". In our own matrix it produced a run that returned 113 of 521 reference words for Norwegian — 22% of the audio — after spending 30 minutes of compute to do it. The text it did produce was correct, aligned sentence for sentence with the reference. It stopped after 9 segments.

Our own pipeline logged that run as COMPLETED. No error, no warning, no partial-result flag. That is the part worth writing down.

What was measured

5 Whisper artifacts from the large-v3 family, across 29 languages, 143 of 145 cells. Audio is digit-free, level-normalised FLEURS — read speech with published reference transcripts — at roughly 460–2,520 units per language.

Every run went through the shipping application on a macOS 15 virtual machine, not through whisper-cli. That distinction matters: it means voice-activity detection, the hallucination filter, loudness normalisation and the language clamp were all in the path, exactly as a user would have them. Language was hard-clamped per run and honoured in all 143 cells; the model actually loaded matched the model requested in all 143 cells. The two absent cells are Hindi on both large-v3 artifacts, which did not complete at all.

Where it happened

Coverage below is the transcript length divided by the reference length — how much of the audio came back:

• Large V3 — 5 of 28: Norwegian 0.22 · Malay 0.32 · Vietnamese 0.34 · Korean 0.40 · Dutch 0.74
• Large V3 Q5 — 4 of 28: Czech 0.22 · Portuguese (Brazil) 0.35 · Vietnamese 0.37 · Malay 0.75
• Large V3 Turbo Q5 — 1 of 29: Hindi 0.72

Every extreme cell in this matrix is short, never long. Not one run in 143 overran its reference by more than half. That is the opposite sign from the repetition-loop failure — the one where a model gets stuck and emits the same phrase for minutes, giving coverage of five times the reference or more. Both are catastrophic and both are invisible to a spot check, but they are distinguishable by the sign of the same number, which is what makes a single check able to catch both.

You can detect this without ground truth

This is the part that transfers. Word error rate needs a reference transcript, which in production you never have. Coverage does not: you already know how long the audio was, and you can see how much transcript came back.

In this matrix the two populations separate cleanly:

The 0.80 threshold is not fitted to the data. The highest coverage among truncated runs is 0.75 and the lowest among healthy runs is 0.82, so there is an empty band between the two populations and any cut inside it gives the same answer. On a corpus without that gap you would need to pick more carefully — and you would be able to see that you had to.

A rough production version: compare the timestamp of the last returned segment against the duration of the file. If a transcript ends at 0:31 of a five-minute recording, that is checkable at runtime, on the user's machine, with nothing to compare against.

Truncation inflates the score it hides in

Because a stopped transcript is scored as though it were a bad one, truncation contaminates the headline number. Large V3 averages 24.64% across all 28 of its languages. Drop the 5 truncated runs and the remaining 23 average 15.25% — so roughly 9.4 points of its error rate is not the model mis-hearing anything. It is the model not being there.

The honest reading cuts both ways: even with truncation excluded, Large V3 does not catch the 834 MB turbo quantisation, which averages 9.16% across all 29 languages and never truncated once. Size did not predict reliability here, and it did not predict accuracy either.

What this says about what we ship

Stated plainly, because it is the reason this page is worth publishing rather than filing:

• Both large-v3 artifacts are offered in the Lesskeys model picker today, and they are the two that truncate. A user who reasons "bigger download, better result" is choosing the least reliable option in the list.
• The 834 MB quantisation that posted the lowest mean error and zero truncations is not in the picker. It was reachable only because the model directory is scanned at startup.
• The shipping default (547 MB) truncated once in 29 languages, on Hindi.
• There is no coverage guard in the product yet. The check described above is a proposal, not a shipped feature. When it ships, this paragraph gets rewritten rather than quietly deleted.

Scope and limits

• Read speech, not meetings. FLEURS is clean recorded prose. Conversational and overlapping audio is materially harder, and these rates should not be read as what you would get on a real call.
• One pass per cell. 143 cells, one run each. Truncation was reproducible across the two large-v3 artifacts and across two separate investigations, but this matrix does not establish a rate — it establishes that it happens.
• One machine. A single Apple Silicon VM at fixed RAM. Decoding was left at the application's file-path defaults rather than the command-line tool's.
• Absolute rates carry some inflation from punctuation and casing normalisation choices. The comparisons between models are the load-bearing part, not the absolute percentages.

The transcripts were kept this time. In an earlier sweep they were overwritten by the next run's configuration step, and truncation had to be inferred from word counts — which is exactly why it went unnoticed for as long as it did. "Worse error rate" and "stopped early" look identical in a results table.

FAQ

Related