Stop Using BLEU to Judge Localization Quality. Here's What to Use Instead.

Why does my MT output score high on BLEU but still need heavy post-editing?

You're not alone. That's the classic localization frustration: machine translation (MT) scores a 0.80 BLEU, your client is happy, then your post-editor spends an hour untangling a single paragraph. BLEU is a blunt instrument, and if you're relying on it to judge localization quality, you're measuring the wrong thing. This article isn't a history lesson—it's a practical guide to picking the right metric for your localization pipeline, and the short answer is: stop using BLEU as your primary quality gate. Instead, measure post-editing effort directly, and if you must use an automatic metric, switch to neural-based ones like COMET or BLEURT.

What BLEU actually measures—and why it lies

BLEU, introduced by IBM researchers in 2002, scores translations on a 0-to-1 scale by comparing n-gram matches between machine output and human reference translations (IBM Research). It was a breakthrough at the time, and it's still the most cited MT metric. But here's the problem: BLEU rewards surface-level similarity to a single reference, not meaning. A translation can be perfectly fluent and accurate yet score poorly because it uses a different word order or a synonym that the reference didn't include. Conversely, a translation can be garbage that happens to overlap with the reference's n-grams and score high.

Worse, BLEU correlates poorly with human judgment—a fact IBM Research itself now acknowledges, noting that newer metrics like COMET and BLEURT are becoming standard (IBM Research). For localization, where the target is a usable, natural-sounding text, not a token-by-token match to a reference, BLEU is dangerously misleading.

The metric that matters: post-editing effort

In localization, the real cost driver is not how "close" the MT output is to some reference; it's how much time a human editor needs to fix it. That's measured by Translation Edit Rate (TER), which counts the number of edits—insertions, deletions, substitutions, and shifts—needed to change the MT output into a reference translation (Snover et al., 2006). A more practical variant, Human-targeted Translation Edit Rate (HTER), has a human create a reference that is closest to the machine output, making it a better proxy for post-editing effort (Snover et al., 2006).

If you can measure HTER, you're measuring the actual labor cost. But HTER requires a human to create a reference, which is expensive. So what do you do when you need an automatic metric that approximates human effort?

Neural metrics: COMET and BLEURT

Enter neural metrics. COMET, developed by Unbabel, is a neural framework that won the WMT 2019 and WMT 2020 metrics shared tasks (Rei et al., 2020). BLEURT, introduced by Google Research in 2020, is BERT-based and correlates better with human judgment than BLEU on several generation tasks (Sellam et al., 2020). Both are trained to predict human quality judgments, so they align with what humans actually care about: fluency, adequacy, and meaning preservation, not n-gram overlap.

In practice, COMET and BLEURT give you a score that reflects whether a human would judge the translation as good, not whether it matches a reference. For localization, that's a far more useful signal. If you're evaluating MT output for a new client or a new language pair, run COMET or BLEURT on a small sample. You'll get a score that tells you how much trouble your post-editors are likely to have.

When BLEU still has a place (maybe)

BLEU isn't useless. It's fast, free, and language-independent. For large-scale, low-stakes screening—say, comparing two MT systems on a test set where you just need a rough ranking—BLEU can be a quick filter. And in research, BLEU is still the common benchmark; the Transformer paper reported a BLEU of 28.4 on English-to-German, which was a big deal (Vaswani et al., 2017). But for localization QA, where you're deciding whether to ship a translation, BLEU is the wrong tool.

Here's a concrete example: You're localizing a medical device manual into German. Google Translate's output scores 0.82 BLEU against a reference. But a post-editor finds that the MT mistranslates a dosage instruction, and fixing it takes 20 minutes. COMET would likely flag that sentence as low-quality because it would detect the semantic mismatch. BLEU wouldn't—it just sees n-gram overlap.

How to choose the right metric for your workflow

If your goal is to reduce post-editing cost, HTER is the gold standard. If you need an automatic metric, use COMET or BLEURT. If you're benchmarking MT research, BLEU is fine. Just don't confuse them.

Make the switch today

Here's my recommendation, plainly: For every localization project, set up a small evaluation set—say, 200 sentences—and run COMET and BLEURT on it. Use those scores to decide whether to use MT raw, post-edit lightly, or post-edit heavily. If you can't run neural metrics, at least use TER on a sample with a human-created reference. And if you must use BLEU, treat it as a rough screening tool, not a quality gate.

This isn't just theory. The language industry is already moving this way. WMT, the annual evaluation campaign, has a metrics shared task that assesses MT quality without a reference using quality estimation (WMT22). And ISO 18587:2017 standardizes full human post-editing of MT output, which means the industry now acknowledges that MT output is not ready to ship without human intervention (ISO 18587:2017).

In short: BLEU is a relic. Stop using it to judge localization quality. Measure post-editing effort or use neural metrics that align with human judgment. Your post-editors—and your clients—will thank you.

Sources

• IBM Research - https://research.ibm.com/blog/bleu-nlp-benchmark-anniversary
• METEOR (CMU) - http://www.cs.cmu.edu/~alavie/METEOR/
• chrF (ACL Anthology) - https://aclanthology.org/W15-3049/
• COMET (ACL Anthology) - https://aclanthology.org/2020.emnlp-main.213/
• Translation Edit Rate (ACL Anthology) - https://aclanthology.org/2006.amta-papers.25/
• BLEURT (Google Research) - https://research.google/pubs/bleurt-learning-robust-metrics-for-text-generation/