Stop Chasing BLEU: Localization Needs Edit Rate, Not Benchmarks

There's a myth floating around localization circles that a higher BLEU score means better machine translation. It doesn't. BLEU is a coarse, n-gram overlap measure that correlates poorly with human judgment. And its continued use has led teams astray, optimizing for the wrong number. The truth? For working translators, the metric that matters isn't BLEU. It's Translation Edit Rate (TER), because it directly approximates the post-editing effort that eats up our hours and budgets.

How BLEU Became the Default

BLEU was introduced by IBM researchers in 2002 as a quick, automatic way to compare machine output against human references (IBM Research). It scores translations from 0 to 1 by counting n-gram matches. Back then, it was a breakthrough—you could suddenly evaluate MT without human judges. But BLEU's convenience masked its limitations. It rewards lexical similarity, not meaning. A translation can be perfectly fluent and accurate yet score low because it uses synonyms or restructures a sentence.

The Localization Reality: Post-Editing Effort

In localization, we don't just evaluate MT in the abstract; we integrate it into a workflow. The dominant model is machine-assisted human translation, where CAT tools like SDL Trados or memoQ leverage translation memories and MT (Machine translation (Wikipedia)). When MT output is used, it inevitably requires human post-editing, especially for idiomatic expressions, cultural nuance, and domain-specific terminology (Machine translation (Wikipedia)). The cost of that post-editing is the real metric. TER measures the number of edits—insertions, deletions, substitutions, and shifts—needed to turn MT output into a reference (Translation Edit Rate (ACL Anthology)). A variant, HTER, even uses a human-created reference closest to the machine output, making it a strong proxy for post-editing effort (Translation Edit Rate (ACL Anthology)).

Why BLEU Misleads Localization Decisions

Here's a scenario: you're choosing between two MT engines for a patent translation project. Engine A scores 45 BLEU; Engine B scores 40. You'd pick A, but you might be wrong. Because BLEU's n-gram matching can be gamed or inflated by superficial similarities. Newer metrics like COMET and BLEURT correlate better with human judgment (IBM Research), but they still don't directly measure the effort a translator will expend. TER does. Studies have shown that BLEU correlates poorly with human judgment, which is why the field is moving toward metrics like COMET and BLEURT (IBM Research). But for localization, the most actionable number is TER.

The Counter-Argument: BLEU Is Quick and Standardized

Some will argue that BLEU is the only metric that's universally understood and cheap to compute. It's true that BLEU requires no human intervention and is still used in research papers and shared tasks like WMT22 (WMT22 (statmt.org)). But the standardization argument is weak. TER is also simple to compute and has been around since 2006 (Translation Edit Rate (ACL Anthology)). Moreover, the WMT metrics shared task itself now evaluates a range of metrics, not just BLEU, and quality estimation tasks are moving beyond reference-based scores (WMT22 (statmt.org)). The industry is evolving, and localization should lead, not lag.

How to Use TER in Practice

Start measuring TER on your own post-editing projects. Use it to compare MT engines against your specific domain, not just generic benchmarks. For example, if you handle legal documents in German, run a sample through Google Translate, DeepL, and a custom engine, and compute TER against a professional reference. The engine with the lowest TER will save you the most money. In one real scenario, a 2,000-word patent text might require 1,000 edits with one engine but only 700 with another—a 30% difference in post-editing time. That's the number that matters.

Bottom Line

Stop optimizing for BLEU. Start tracking Translation Edit Rate. It's the only metric that directly reflects the cost of human post-editing, which is the heart of localization. When you're evaluating MT for your workflow, ask not “What's the BLEU score?” but “How many edits per 100 words will my translators make?” That's the question that saves you money and sanity.

Sources

• Machine translation - Wikipedia - https://en.wikipedia.org/wiki/Machine_translation
• IBM Research - BLEU - https://research.ibm.com/blog/bleu-nlp-benchmark-anniversary
• Translation Edit Rate - ACL Anthology - https://aclanthology.org/2006.amta-papers.25/
• WMT22 - statmt.org - https://www.statmt.org/wmt22/
• ISO 18587:2017 - ISO - https://www.iso.org/standard/62970.html