BLEU Scores Are Lying to You. Here's What to Use Instead.

Still Using BLEU? Let's Talk About What You're Actually Measuring

Look at any machine translation paper or vendor demo and you'll see a single number: the BLEU score. It's supposed to tell you how good a translation is. But here's the thing—it often doesn't. BLEU was cooked up by IBM researchers back in 2002 as a quick, cheap way to compare systems. It works by counting how many word sequences (n-grams) in the machine output match a human reference translation (IBM Research). That might sound reasonable, but it's been shown time and again to correlate poorly with what humans actually think is good (IBM Research).

So why is it still everywhere? Because it's easy to compute. But easy isn't the same as useful. When you're deciding which translation tool to buy or which output to publish, you need a metric that reflects real quality—not just string matching.

What BLEU Gets Wrong: A Concrete Example

Imagine you're translating an English sentence into French: "The cat sat on the mat." A good translation might be "Le chat s'est assis sur le tapis." But BLEU counts n-gram overlaps, so if the machine outputs "Le chat était assis sur le tapis" (which is also perfectly fine), it gets penalized for not matching the reference exactly. Meanwhile, a translation like "Le chat assis sur le tapis était" might get a higher BLEU score because it shares more word chunks—even though it's grammatically awkward and a human would immediately flag it.

That's not a theoretical edge case. It's a known flaw. BLEU was designed to compare systems quickly, not to judge quality. And the consequences are real: a system with a high BLEU score can still produce output that reads like garbage. That's why newer metrics like COMET and BLEURT are gaining traction—they correlate much better with human judgment (IBM Research).

The Counter-Argument: "But BLEU Is All We Have"

You might be thinking, "Okay, BLEU is imperfect, but it's cheap, fast, and works for any language. What else is out there?" That's a fair point, but it's not true anymore. There are metrics that are both better and practical.

Take chrF, for instance. It's a character n-gram F-score proposed by Maja Popović in 2015 (chrF). Because it works on characters rather than words, it's language- and tokenization-independent—perfect for morphologically rich languages like Turkish or Finnish, where word-level matching fails. Or consider COMET, a neural framework from Unbabel that won the WMT metrics shared task in 2019 and 2020 (COMET). COMET uses neural networks to score translations based on meaning, and it correlates far better with human judgment than BLEU ever did. Then there's BLEURT, a BERT-based metric from Google Research that also beats BLEU on human correlation (BLEURT).

These aren't pie-in-the-sky ideas. They're available now, and you can run them on your own data. The only reason to stick with BLEU is inertia—and inertia is a terrible reason to make translation decisions.

What to Use Instead: Edit Distance and Human Effort

My recommendation: stop chasing a single number and start measuring what you actually care about—human effort. The most practical metric for real-world translation is Translation Edit Rate (TER). TER measures how many edits (insertions, deletions, substitutions, and shifts) are needed to change the machine output into a correct reference translation (Translation Edit Rate). Even better, use Human-targeted TER (HTER), where a human creates the reference that's closest to the machine output, giving you a realistic proxy for post-editing effort (Translation Edit Rate).

That number tells you how much time a translator will spend fixing the output. And if you're following ISO 18587, the standard for post-editing machine translation, you already know that human post-editing is an essential step (ISO 18587). So why not evaluate the output the way you'll actually use it?

Here's a concrete scenario: You're translating a user manual from English to German. System A gets a BLEU score of 45—sounds decent. But when you calculate HTER, you find that a translator needs to make 30 edits per 100 words. That's a lot of manual work. System B has a BLEU score of 40 but an HTER of 15. Which one would you choose? BLEU would have led you astray.

The Future Is Neural, So Your Metric Should Be Too

Machine translation has evolved from rule-based to statistical to neural. Today, models like the Transformer power everything from Google Translate to DeepL (Attention Is All You Need). The Transformer achieved a BLEU score of 28.4 on English-to-German, but that's beside the point. The point is that modern neural systems are fluent enough that n-gram matching is even less useful.

You need metrics that understand meaning, like COMET or BLEURT. And don't just take my word for it—look at the WMT evaluation campaigns. WMT22, for example, organized a metrics shared task to assess MT quality with or without a reference, and a quality estimation task that assesses quality without any reference at all (WMT22). That's where the field is heading. So if you're serious about translation quality, it's time to move beyond BLEU.

Sources

• IBM Research - https://research.ibm.com/blog/bleu-nlp-benchmark-anniversary
• chrF (ACL Anthology) - https://aclanthology.org/W15-3049/
• COMET (ACL Anthology) - https://aclanthology.org/2020.emnlp-main.213/
• BLEURT (Google Research) - https://research.google/pubs/bleurt-learning-robust-metrics-for-text-generation/
• Translation Edit Rate (ACL Anthology) - https://aclanthology.org/2006.amta-papers.25/
• WMT22 (statmt.org) - https://www.statmt.org/wmt22/