Forget BLEU: Why Machine Translation Metrics Are Fooling You

For years, we’ve been told that BLEU is the gold standard for evaluating machine translation. I’m here to tell you it’s time to retire it. The metric that helped launch the modern MT era has outlived its usefulness, and clinging to it is actively hurting your translation quality. In this post, I’ll explain why BLEU is a relic, what to use instead, and why the human translator remains irreplaceable.

The BLEU Delusion

BLEU, introduced by IBM researchers in 2002, was a breakthrough: it automatically compared machine output to human references by counting n-gram overlaps (IBM Research). But it’s a crude proxy. It rewards literal matches, not meaning. A translation that captures the sense but uses different wording gets penalized, while a word-for-word rendering that sounds like a robot often scores higher. IBM itself now admits that BLEU correlates poorly with human judgment (IBM Research). Yet we still see BLEU scores cited in papers, blogs, and product marketing. It’s time to stop.

What to Use Instead

The good news is that better metrics exist. If you want a quick, language-independent check, chrF, which compares character n-grams, is a solid choice (Popović, 2015). For a more nuanced evaluation, COMET, a neural framework from Unbabel, consistently outperforms BLEU in correlation with human judgment (Unbabel). BLEURT, from Google, is another neural option that’s shown promise (Google Research). And if you’re measuring post-editing effort, Translation Edit Rate (TER) tells you how many edits a human would need to make (Snover et al., 2006). But here’s the catch: even these metrics are imperfect. They’re all proxies for the real judge: a human being.

The Human Factor

You might argue that metrics are objective and scalable, while humans are slow and subjective. Fair point. But consider this: the ISO 17100:2015 standard, which governs translation services, requires human review, not just automated scoring (ISO). And ISO 18587:2017 specifically addresses post-editing of MT output, acknowledging that human post-editing is essential for quality (ISO). Why? Because MT, especially neural MT, still struggles with nuance. DeepL, which launched in 2017, was reportedly chosen by professional translators three times more often than Google, Microsoft, or Facebook in blind tests (DeepL). That’s a human preference, not a metric. So yes, metrics are convenient, but they can’t replace the translator’s judgment.

The Bottom Line

Stop obsessing over BLEU. If you’re evaluating MT systems, use a mix of COMET or BLEURT for overall quality, TER for post-editing effort, and always, always run a human check. For a quick sanity test, chrF is a fine lightweight option. But remember: no metric can tell you if a translation reads naturally or captures the cultural context. That’s your job.

Quick tip: When comparing MT engines, don’t just look at the average score. Test with your own domain-specific texts and have a professional translator rate the outputs. The metric that matters most is the one that reflects your actual use case.

What I’d Actually Do

If I were setting up an MT evaluation pipeline today, I’d use COMET as my primary metric, supplemented by TER for post-editing cost. I’d also keep a small human evaluation panel—even just three bilingual speakers—to rate a random sample of outputs. That combination gives you the scalability of automated metrics and the reliability of human judgment. And I’d ditch BLEU entirely, except maybe for historical comparisons. Trust me, your translations will thank you.

Sources

• IBM Research - BLEU blog - https://research.ibm.com/blog/bleu-nlp-benchmark-anniversary
• chrF paper (ACL Anthology) - https://aclanthology.org/W15-3049/
• COMET paper (ACL Anthology) - https://aclanthology.org/2020.emnlp-main.213/
• TER paper (ACL Anthology) - https://aclanthology.org/2006.amta-papers.25/
• BLEURT (Google Research) - https://research.google/pubs/bleurt-learning-robust-metrics-for-text-generation/
• ISO 17100:2015 - https://www.iso.org/standard/59149.html