LesskeysWhen Whisper stops early and says it finished
A Whisper model can stop partway through a recording and still report success: 10 of 143 runs in our test did. Comparing how much transcript came back with how long the audio was catches it, with no reference transcript.
macOS 13+ on Apple Silicon · No subscriptionWindows 10 and 11 · No subscription
Last reviewed
What it is
What silent truncation is
The text it did produce was correct. It just stopped, and the pipeline said it was done.
A speech model can return the wrong words, or it can stop partway and hand back a correct transcript of the first part as though it were finished. In our worst run, Norwegian, it returned 113 of 521 reference words after 30 minutes of compute, and stopped after 9 segments. Our own pipeline logged that run as COMPLETED: no error, no warning, no partial-result flag.
- 10 of 143runs stopped early
- 22%of the audio in the worst run
- 9 of 10on the two large-v3 files
Which runs
Which languages stopped early
9 of the 10 are on the two large-v3 files.
Coverage is the transcript length divided by the reference length: 1.00 is a full transcript. Each bar is one truncated run; the dashed line is the 0.80 cut-off.
Large V3: 5 of 28 runs
- Norwegian0.22
- Malay0.32
- Vietnamese0.34
- Korean0.40
- Dutch0.74
Large V3 Q5: 4 of 28 runs
- Czech0.22
- Portuguese (Brazil)0.35
- Vietnamese0.37
- Malay0.75
Large V3 Turbo Q5: 1 of 29 runs
- Hindi†0.72
Hindi: a scoring caveat
† This is the default model's run: 886 units against a reference of 1,237, the same inflated count that the correction on Whisper accuracy in 29 languages names (Hindi: 1,237 units against 683 real words, from a scorer that split words at combining marks), so the absolute 0.72 is not reliable. But the two turbo siblings scored 1,288 and 1,242 units against the same reference (coverage 1.04 and 1.00), so the default's run was 29 to 31 percent shorter than theirs. The most likely reading is a real truncation, which is also how the job's own results note records it. It ran in the 12 GB part of the matrix (see the limits below).
Short, never long
Every extreme cell in this matrix is short, never long. Not one run in 143 overran its reference by more than half. That is the opposite sign from the repetition-loop failure, where a model gets stuck and emits the same phrase for minutes, giving coverage of five times the reference or more. Both are catastrophic and both are invisible to a spot check, but they are distinguishable by the sign of the same number, which is what lets a single check catch both.
Catch it
Catch it without a reference
Compare how much transcript came back with how long the audio was.
Word error rate needs a reference transcript, which in production you never have. Coverage does not. In this matrix the two populations separate cleanly:
| What | Runs | Mean coverage | Mean error rate |
|---|---|---|---|
| Truncated | 10 | 0.44 | 65.90% |
| Healthy | 133 | 1.01 | 12.10% |
The 0.80 cut-off is not fitted to the data: the highest coverage among truncated runs is 0.75 and the lowest among healthy runs is 0.82, so any cut inside that empty band gives the same answer. In production: compare the timestamp of the last returned segment against the duration of the file, on the user's machine, with nothing to compare against.
The error rate
Truncation inflates the error rate
Large V3 averages 24.64% across 28 languages; without its 5 truncated runs, 15.25%.
Roughly 9.4 points of its error rate is not the model mis-hearing anything. It is the model not being there.
Even without truncation, Large V3 does not catch the smaller turbo model
Even with truncation excluded, Large V3 does not catch the 834 MB turbo quantisation, which averages 9.16% across all 29 languages and never truncated once. On the 28 languages that all three turbo runs share (Hindi set aside), the means are Q8 8.59%, the default 8.71% and the 1.6 GB turbo 8.80%, within 0.2 points of each other, so most of the headline gap is one Hindi row. Size did not predict reliability here, and it did not predict accuracy either.
On a Mac
What this means for the models Lesskeys offers
Both large-v3 files are in the model picker of the App Store version on sale (3.1.4), and they account for 9 of the 10 truncated runs.
There is no coverage check in that version. The check described above is a proposal for it, not a shipped feature.
Status on 15 October 2026, for the App Store version on sale (3.1.4)
Stated plainly, because it is the reason this page is worth publishing rather than filing:
- Both large-v3 artifacts are offered in the Lesskeys model picker, and they account for 9 of the 10 truncated runs. A user who reasons “bigger download, better result” is choosing the least reliable option in the list.
- The 834 MB quantisation that posted the lowest mean error (a gap to the other turbo runs that is mostly one Hindi row) and zero truncations is not in the picker of that version. It was reachable in this test only because the model directory is scanned at startup.
- The default model (547 MB) truncated once in 29 languages, on Hindi (most likely a real truncation; see the note under the coverage list). That run and the Czech Large V3 Q5 run both ran in the 12 GB part of the matrix, with little memory to spare (see the limits).
- There is no coverage check in that version. The check described above is a proposal for it, not a shipped feature. When it ships, this section gets rewritten rather than quietly deleted.
On Windows
What this means on Windows
The test was run in the Mac app. The Windows app was not part of it.
When speaker labels are on, Lesskeys on Windows has a safety net for a transcript that comes back empty from a file with speech in it: it decodes the file again without timestamps. It has no check for a transcript that stops partway, which is the case on this page. On a long file, read the last lines of the transcript against the audio, and check that the end of the transcript matches the end of the recording.
The data
The method, the table and the limits
How the runs were made and what they cannot tell you, collapsed.
How the truncation was measured
5 Whisper artifacts from the large-v3 family, across 29 languages, 143 of 145 cells. The audio is level-normalised FLEURS with no digits in the text (read speech with published reference transcripts) at roughly 460 to 2,520 units per language.
Every run went through the shipping application on a virtual machine running macOS Sequoia, not through whisper-cli. That matters: voice-activity detection, the hallucination filter, loudness normalisation and the language clamp were all in the path, exactly as a user would have them. Language was hard-clamped per run and honoured in all 143 cells; the model actually loaded matched the model requested in all 143. The two absent cells are Hindi on both large-v3 artifacts, which did not complete at all.
The five models: size, mean error, truncated runs
Five large-v3-family Whisper artifacts across 29 languages (rows with no truncated run are highlighted)
| Model | Size | Languages | Mean error | Median | Truncated runs |
|---|---|---|---|---|---|
| Large V3 Turbo Q8 ● not in the picker | 834 MB | 29 | 9.16% | 6.80% | none |
| Large V3 Turbo | 1.6 GB | 29 | 9.52% | 7.23% | none |
| Large V3 Turbo Q5 ● shipping default | 547 MB | 29 | 9.99% | 7.81% | 1 of 29 |
| Large V3 | 3 GB | 28 | 24.64% | 15.80% | 5 of 28 |
| Large V3 Q5 | 1.0 GB | 28 | 26.66% | 18.66% | 4 of 28 |
The limits of this test
- Read speech, not meetings. FLEURS is clean recorded prose. Conversational and overlapping audio is materially harder, and these rates should not be read as what you would get on a real call.
- One pass per cell. 143 cells, one run each. Truncation was reproducible across the two large-v3 artifacts and across two separate investigations, but this matrix does not establish a rate; it establishes that it happens.
- One machine, two memory sizes. The cells ran on one Apple Silicon virtual machine whose memory was 12 GB for the first 33 cells and 24 GB for the rest; all 29 default-model cells and the first four Large V3 Q5 cells (Arabic, Czech, Danish, German) ran in the 12 GB part. Two truncated runs fall in it: Hindi on the default (241 MB of memory available at the lowest point) and Czech on Large V3 Q5 (60 MB). Every other truncated run had more than 6,000 MB available. Low memory could have contributed to those two, and this matrix cannot separate that from the model. Decoding was left at the application's file-path defaults rather than the command-line tool's.
- Absolute rates carry some inflation from punctuation and casing normalisation choices. The comparisons between models are the load-bearing part, not the absolute percentages.
The transcripts were kept this time. In an earlier sweep they were overwritten by the next run's configuration step, and truncation had to be inferred from word counts, which is exactly why it went unnoticed for as long as it did. “Worse error rate” and “stopped early” look identical in a results table.
Questions
Questions about Whisper stopping early
What is silent truncation in speech recognition?
A model returns a transcript covering only part of the audio and the surrounding pipeline reports success. The text that is present is usually correct, so nothing looks wrong until someone compares the transcript against the recording. In this matrix it affected 10 of 143 runs, 9 of them on the two large-v3 artifacts.
How can I detect a truncated transcript without a reference?
Compare what came back against what went in: the end timestamp of the last segment versus the duration of the audio file, or transcript length versus expected length. Neither needs a ground-truth transcript, which is what makes the check usable in production. In this data, truncated runs averaged 0.44 coverage against 1.01 for healthy ones.
Does a bigger Whisper model give better results?
Not in this matrix. The two largest artifacts posted the highest mean error rates and accounted for 9 of the 10 truncated runs, while a 834 MB quantisation of the turbo model posted the lowest mean (9.16%, within 0.2 points of the other turbo runs once Hindi is set aside) and never truncated. Download size predicted neither accuracy nor reliability here.
Is this a bug in Whisper or in the app?
Undetermined, and the page does not claim to know. What is measured is that the condition occurs and that the surrounding pipeline did not notice it. The proposed coverage guard is a mitigation at the application layer, which is the layer we control; it does not diagnose the underlying cause.
More