Word Error Rate (WER) in Speech-to-Text: What It Measures, Where It Falls Short, and Which Benchmarks to Trust

Why WER Matters

Word Error Rate (WER) is the standard for measuring the accuracy of automatic speech recognition (ASR) and speech-to-text transcription. It counts the words a system gets wrong compared with a correct reference transcript, which has historically made it a simple, objective way to compare systems and track progress over time.

It still gives you a fast, reproducible, directional number, and it is the figure reported by major leaderboards and nearly every vendor. But a single WER figure treats every error alike, depends entirely on the reference transcript and scoring rules behind it, and says nothing on its own about the audio it was measured on, which is why many today use it as a starting reference point when evaluating transcription tools, and not as the sole basis for choosing one.

This page explains what WER measures, why it can be misleading, how to read the public benchmarks for speech-to-text accuracy, and how to run a transcription comparison on your own audio.

Why WER Often Fails in Production

Since WER is still the most widely used measure of transcription accuracy, its blind spots matter. Outside of clean, single-speaker audio, it often breaks down in three ways once speech-to-text models run production audio:

1. Not all errors cost the same. WER counts dropped filler words exactly the same as misheard business-critical terms such as a medication dosage, an account number, or a proper name. The real cost also depends on the application: in a voice agent, a plausible wrong guess can keep a conversation moving while a missed word can leave the agent with nothing to respond to, and a transcript passed to an LLM or analytics pipeline tolerates different errors than one read by a person.
2. The reference transcript is not always right. WER measures distance from a human-written reference and human transcribers routinely skip overlapping speech, false starts, and disfluencies. When a model correctly transcribes words that the reference omitted, the metric scores them as insertion errors, so a more accurate system can score worse. Artificial Analysis, an independent benchmarker, found errors in the references for two widely used public test sets (VoxPopuli and Earnings-22) and built corrected versions in its February 2026 benchmark update. Across the models it tested, average WER fell by 3.5 percentage points on VoxPopuli and 5.6 on Earnings-22 once the references were corrected.
3. Formatting differences register as errors. "Healthcare" and "health care," "alright" and "all right," and "gonna" and "going to" are all acceptable renderings, but a strict word-by-word comparison scores them as mistakes unless normalization is applied first. Normalization can change results substantially (see below), and models differ in how they format output by default.

One note on provenance: the most visible 2026 argument that "WER is broken" came from AssemblyAI, a vendor with its own models to promote. The underlying concerns are not vendor-specific. Artificial Analysis corrected the reference data itself, and Sierra, a company that deploys voice agents rather than selling transcription, built a benchmark in April 2026 on the premise that WER alone is misleading. When a vendor, an independent benchmarker, and a buyer reach similar conclusions from different directions, the conclusion deserves weight.

What WER Actually Measures

WER expresses the number of errors a system makes relative to the number of words in a reference transcript:

WER = (Substitutions + Deletions + Insertions) / Total Reference Words

• Substitutions: the system writes a different word than the one spoken
• Deletions: the system misses a word that was spoken
• Insertions: the system adds a word that was not in the reference

Worked example. Reference: "the cat sat on the mat" (6 words). System output: "the cat sat on a mat today." One substitution ("the" became "a") and one insertion ("today") produce 2 errors across 6 reference words, a WER of 33%.

Because the formula divides by the reference length, WER can exceed 100% when a system inserts many extra words, and a sloppy or incomplete reference distorts every score computed against it.

Several details are left open by the formula, and vendors do not always resolve them the same way:

• Normalized vs. raw. Normalization standardizes casing, punctuation, numerals, contractions, and filler words before scoring. The effect can be large: in one academic comparison of commercial transcription vendors, average WER across datasets fell from roughly 21.8% with no normalization to about 11.3% after lowercasing and removing punctuation. Artificial Analysis, for example, lowercases text and strips bracketed text and filler words before scoring, which is a different rule set than another evaluator might apply.
• Word-level vs. character-level. WER suits space-delimited languages like English. For languages such as Mandarin or Japanese, evaluators typically report Character Error Rate (CER) instead, and the two are not interchangeable.
• Chunked vs. full-call scoring. Long recordings may be split into segments and reassembled before scoring, or scored segment by segment. Results can differ.
• Scoring tool. One study found that different open-source scoring and normalization libraries produced measurably different WER on the same transcripts.

Before treating any published WER as meaningful, confirm which of these choices produced it.

Why WER Numbers Are Hard to Compare Across Vendors

Even if every vendor reported WER honestly, a few factors still make headline WER figures difficult to compare:

Different test sets. WER is only meaningful relative to the audio it was measured on. A system tested on clean-read speech and one tested on noisy contact-center audio can each report an honest figure, and the two figures still say nothing about which will perform better on your audio.

Self-reported vs. independently measured. A vendor chooses its own test set, model version, and configuration. An independent evaluator tests many vendors on the same audio under the same rules. The two kinds of numbers answer different questions, and a vendor's own comparison and an independent leaderboard can rank the same providers differently because they use different audio, reference transcripts etc.

Reference quality. As covered above, errors in the reference transcript are scored as errors by the system, in either direction.

Domain vocabulary. Medical, legal, financial, and product-specific terminology raises WER for every vendor, but not equally, depending partly on how well a vendor's training data covers the domain.

Accent, dialect, and language. Most public benchmarks are English-centric, and those that include other languages often rely on read speech recorded in quiet studios. Accuracy can vary dramatically across languages and accents in ways an English benchmark never surfaces.

Audio conditions. Telephony compression (phone calls are commonly recorded at 8 kHz), background noise, poor microphones, and overlapping speakers all raise WER, and vendor benchmark audio is frequently cleaner than production audio. Noisy or silent segments can also trigger hallucinated text in some models, which shows up as insertions.

Snapshot timing. Models are updated frequently, and a score reflects one model version on one date.

These factors do not make benchmark numbers meaningless, but they do make them hard to digest without their methodology. Some of the most useful comparisons come from a single evaluator testing multiple vendors under identical conditions. The most reliable comparison is the one you run on your own audio.

Public Benchmarks Worth Knowing

A handful of public resources do most of the work of comparing transcription systems independently. Each answers a different question, and none is a complete picture on its own.

Hugging Face Open ASR Leaderboard

The Hugging Face Open ASR Leaderboard is a leaderboard, not a dataset. It scores submitted models, including commercial APIs, against standardized public test sets under a single methodology. It covers short-form English, long-form English (earnings calls, TED talks, and similar), and a multilingual track.

Two things matter when reading it. First, rankings are specific to a track, a set of test sets, and a date. A "#1" is a claim about one track at one moment, and the number of models ranked changes as submissions arrive. Near the top of the table, small WER differences can move a model several places, and one team's account of fine-tuning toward the leaderboard reported climbing ten positions by improving on a single test set. Second, because the underlying test sets are public, models can be trained on or tuned to them, which flatters the scores. Keep this in mind when you're reading leaderboard data like this.

Artificial Analysis Speech-to-Text AI Model & Provider Leaderboard

The Artificial Analysis speech-to-text leaderboard publishes independent, side-by-side results for transcription providers on accuracy, speed, and price. Its AA-WER v2 index is a duration-weighted average across roughly eight hours of audio from three datasets: AA-AgentTalk (50%), a proprietary set focused on speech directed at voice agents; VoxPopuli-Cleaned-AA (25%); and Earnings22-Cleaned-AA (25%). The cleaned sets are corrected reference transcripts for public data, and the methodology is published.

Its strengths are independence and a published methodology. Its limits are that AA-AgentTalk is held out and cannot be independently inspected, that the weighting reflects a judgment about voice-agent use cases, and that the benchmark is mostly English. Like any leaderboard, it is a dated snapshot.

The Test Sets Underneath

Two of the most widely used public test sets sit under both leaderboards, so it helps to know what they represent.

• VoxPopuli: European Parliament recordings from 2009 to 2020. The English portion comes largely from non-native English speakers, which makes it a useful signal for accent robustness rather than performance on typical native conversation.
• Earnings-22: A 119-hour corpus of English-language earnings calls from companies worldwide, featuring speakers of many nationalities and accents along with financial vocabulary. It is a reasonable proxy for domain-specific, international business speech.

Because Artificial Analysis and the Hugging Face leaderboard draw on overlapping datasets, they should not be read as two fully independent confirmations of the same result. The Hugging Face leaderboard's evaluation tooling can also be run against Artificial Analysis's cleaned Earnings-22, which means scores depend on which version of the reference is used.

Multilingual and Phone Audio: Sierra μ-Bench

English-only benchmarks leave a gap, and Sierra's μ-Bench is one of the few public attempts to fill it. Released by Sierra in April 2026, it contains 4,270 human-annotated utterances from 250 real phone conversations recorded at 8 kHz mono, across five locales. Sierra evaluated Deepgram Nova-3, Google Chirp-3, Microsoft Azure Speech, ElevenLabs Scribe v2, and OpenAI GPT-4o Mini Transcribe, and reported three headline findings:

• WER alone can be misleading, because it cannot separate meaning-changing errors from surface-level ones
• No provider won everywhere, and the best choice varied by language
• Mandarin transcription accuracy was as much as five times worse than English for some providers, and Vietnamese results varied widely

Sierra operates the benchmark as an open leaderboard and accepts submissions from other providers. Few things to note when taking a look at their leaderboard.

• The benchmark covers five providers and five locales, so it says nothing about providers it did not test.
• The calls are scripted interactions with a banking voice agent, which is realistic for customer service but narrower than open conversation.
• The dataset is access-gated and licensed for transcription-quality testing only.
• And, as Sierra notes, standard text normalization can break down for some languages, which affects any multilingual WER comparison.

The takeaway for buyers is not to rely on a single ranking, but to look at the pattern - provider performance varies by language, and a provider that leads on English may not lead elsewhere. If you serve more than one language, evaluate each language separately.

Vendor-Published Benchmarks

Vendors also publish their own accuracy pages and comparison tables. These are useful for understanding how a vendor measures itself and for spotting what it chooses to emphasize. Keep in mind these are self reported. The vendor selects the datasets, the competitors included, the model versions, and the scoring rules. Treat them as one input, and look for the vendor's stated methodology.

How to Benchmark WER Yourself

Public benchmarks narrow the field. Only your own production audio decides the outcome.

1. Assemble representative audio. Collect recordings across your real conditions: speakers, accents, noise levels, channel quality (phone vs. headset), and specialized vocabulary. If you serve multiple languages, build a separate set per language.
2. Create and audit reference transcripts. Use human transcription, then review disagreements against the audio rather than assuming the reference is correct. Reusing a corrected reference across vendors keeps the comparison fair.
3. Normalize identically for every system. Apply the same casing, punctuation, numeral, and filler-word rules to every transcript. Otherwise you are measuring formatting choices rather than accuracy.
4. Score with one consistent tool. Open-source libraries such as jiwer compute WER (and CER) from a reference and hypothesis pair. Use the same library and settings for every vendor.
5. Report per-condition results. Break out results by language, speaker count, noise level, and channel. A single blended number hides exactly the variation you are trying to find.
6. Validate in production. Static benchmarks do not capture business outcomes. If possible, route a small share of live traffic to a candidate and track downstream measures such as correction rates, task completion, and support volume.

How to Reduce WER

Lowering WER starts with finding where the errors come from. Most of the gains you make in cutting word error rate come from a handful of practical levers rather than from switching vendors alone.

• Diagnose first. Break errors into substitutions, deletions, and insertions, and by condition: language, noise level, speaker, and channel. A high insertion count points to a different fix (noise, silence, hallucinated text) than a high substitution count (vocabulary, accents).
• Start with the audio. Background noise, poor microphones, heavy compression, and overlapping speakers drive errors. Capture and send audio in the quality and format your provider's documentation specifies. Be cautious with noise reduction: one vendor reports that filters built for human listening can make recognition worse, so test any preprocessing against your own audio.
• Match the model to your conditions. Choose by language, accent, domain, and channel (phone versus headset), using results on audio like yours rather than headline numbers.
• Supply your vocabulary. Names, product terms, and jargon are common error sources. Many providers support custom vocabulary or term boosting, and one 2026 study of context-biasing methods built on Whisper reported cutting WER on the boosted words by up to 88% relative while leaving other words largely unaffected.
• Handle non-speech audio. Silence and noise can trigger hallucinated text, and researchers have shown that post-processing filters can reduce the resulting WER.
• Check the scoring. Some counted errors are formatting differences. Apply consistent normalization before concluding one system is worse than another.

FAQ

Conclusion

Word Error Rate is a real and widely-used and accepted metric, but a single WER figure tells a buyer very little without the audio, reference transcript, and scoring rules behind it. The metric treats all errors alike, depends on the quality of a human reference that is itself imperfect, and shifts with normalization choices. Independent resources such as the Hugging Face Open ASR Leaderboard, Artificial Analysis, and Sierra's μ-Bench give buyers a much better starting point than any vendor's own claim, and μ-Bench in particular is a reminder that English results do not carry over to other languages or to phone audio.

They remain a starting point. The comparison that matters most is the one you run on your own recordings, in your own languages, with the same scoring rules applied to every vendor.

Sources used (tier in parentheses)

• AssemblyAI, "Word error rate is broken" (Apr 14, 2026): assemblyai.com/blog/word-error-rate-is-broken (vendor marketing, tier 2). Used for the framing of the three failure modes only, rewritten; no AssemblyAI figures or metrics are used.
• Artificial Analysis STT methodology: artificialanalysis.ai/speech-to-text/methodology; AA-WER v2.0 (Feb 18, 2026): artificialanalysis.ai/articles/aa-wer-v2 (independent benchmarker)
• Sierra μ-Bench blog (Apr 20, 2026): sierra.ai/blog/mu-bench-an-open-multilingual-transcription-benchmark; dataset card: huggingface.co/datasets/sierra-research/mu-bench
• Hugging Face Open ASR Leaderboard: github.com/huggingface/open_asr_leaderboard; datasets: huggingface.co/datasets/open-asr-leaderboard/datasets; paper: arxiv.org/abs/2510.06961; issue #153: github.com/huggingface/open_asr_leaderboard/issues/153
• Academic: arxiv.org/abs/2608.05759 (context biasing); arxiv.org/abs/2603.05267 (Interspeech survey); arxiv.org/abs/2408.16287 (normalization effect); arxiv.org/abs/2408.15616 (scoring tools); arxiv.org/abs/2501.11378 and 2504.05702 (Whisper hallucinations); arxiv.org/abs/2405.06150 (disfluent speech bias)
• Practitioner/vendor explainers consulted for question themes (tier 2, not cited as fact): Deepgram, Gladia, ElevenLabs, Speechmatics, Picovoice, Smallest.ai, Vatis
• Modulate press release (Jul 13, 2026), customerthink.com / streetinsider.com mirrors (vendor press release, tier 2)