Stop Believing the BLEU Score: What Translation Theory Gets Wrong

Imagine you’ve just finished a machine translation project. The BLEU score is 0.87—impressive, right? But when you actually read the translated text, it’s clunky, awkward, and misses cultural nuances. You’re furious. What’s the point of a metric that doesn’t reflect quality? This is the dirty secret of translation theory: the metrics we cling to are often misleading.

You don’t need to be a linguist to see that machine translation has changed the game. But you do need to understand the theory behind the tools, or you’ll make costly mistakes. In this article, I’ll debunk common myths, compare the options, and give you blunt advice on how to actually evaluate translation quality.

Is Machine Translation Really 'Artificial Intelligence'?

Yes, but not in the way you think. Neural machine translation (NMT) uses a large neural network to analyze whole sentences, not just word-by-word (Machine translation, Wikipedia). It’s trained end-to-end on massive parallel corpora. That’s real AI, but it’s not sentient. It’s pattern recognition on steroids. So when someone tells you their translation is “AI-powered,” they’re not lying—they’re just overselling. The quality still depends on the data it was trained on.

What’s the Difference Between MT and CAT Tools?

This is a classic confusion. MT is fully automated—it replaces the human translator during the translation phase. CAT tools (computer-assisted translation) are aids for human translators, giving them translation memories and glossaries (Machine translation, Wikipedia). A translation memory is a database of previously translated segments that can be reused (Machine translation, Wikipedia). In short: MT does the work for you; CAT helps you do the work yourself. You need both, but for different reasons.

Is a High BLEU Score the Gold Standard? (Spoiler: No)

BLEU (Bilingual Evaluation Understudy) was introduced by IBM researchers in 2002 (IBM Research). It compares n-gram matches between machine output and human references, giving a score from 0 to 1. But here’s the kicker: BLEU correlates poorly with human judgment (IBM Research). That’s why better metrics like COMET and BLEURT are becoming standard (IBM Research). COMET is a neural framework that won WMT 2019 and 2020 metrics tasks (COMET, ACL Anthology). BLEURT is BERT-based and correlates better with human judgment (BLEURT, Google Research). So stop obsessing over BLEU and start using metrics that actually mean something.

What’s the Best Way to Evaluate Translation Quality?

There’s no one-size-fits-all answer, but I’ll give you a rule of thumb. For a quick, cheap check, use chrF—it’s language-independent and works well for morphologically rich languages (chrF, ACL Anthology). For serious projects, use a combination: COMET for automatic scoring, and human evaluation for final sign-off. And if you’re post-editing, look at Translation Edit Rate (TER), which measures how many edits you need to make (Translation Edit Rate, ACL Anthology). HTER, where a human creates a reference closest to the machine output, is a good proxy for post-editing effort (Translation Edit Rate, ACL Anthology).

Do I Need a Human Translator at All?

Yes. MT output often requires human post-editing, especially for idiomatic expressions, cultural nuance, and domain-specific terminology (Machine translation, Wikipedia). ISO 18587:2017 even specifies requirements for full human post-editing and post-editors’ competences (ISO 18587:2017, ISO). The EU’s Directorate-General for Translation, one of the largest translation services in the world, still relies on human translators for all 24 official languages (European Commission translation department). Don’t be fooled by the hype—machines aren’t replacing humans yet.

Are NMT Systems Like DeepL Really Better Than Google?

DeepL claims that in blind tests, professional translators chose its translations roughly three times more often than Google, Microsoft, or Facebook (DeepL Translator, Wikipedia). But take that with a grain of salt—it’s their own test. What matters is the use case. For patent documents, WIPO Translate is specifically trained for patents (WIPO Translate, WIPO). For EU institutions, eTranslation is free and covers all 24 EU languages (eTranslation, European Commission). So the “best” system depends on your domain. Don’t just pick the one with the best marketing.

What About Zero-Shot Translation?

Google’s multilingual NMT system, announced in 2016, enabled zero-shot translation—like translating between Japanese and Korean even though it was never trained on that pair (Zero-Shot Translation, Google Research). That sounds magical, but it’s not perfect. It works because the model learns an internal “interlingua” (Zero-Shot Translation, Google Research). Still, for rare language pairs, you’re better off using a model like Meta’s NLLB-200, which supports 200 languages and improves quality by about 44% on average compared to previous systems (No Language Left Behind, Nature).

So, What Should You Actually Do?

Stop chasing the latest shiny metric and focus on the task at hand. If you’re a translator, learn to use CAT tools effectively—they’ll boost your consistency and efficiency. If you’re a project manager, use a mix of automatic metrics and human review. And if you’re a business owner, be skeptical of vendors who quote BLEU scores as proof of quality. They’re gaming you.

Quick tip: For a quick sanity check, run your translation through a different MT engine and compare—if they agree, it’s probably decent; if they diverge wildly, investigate.

What I’d Actually Do

Here’s my blunt recommendation: For anything that will be read by humans, never rely solely on raw MT. Use a CAT tool with a good translation memory, run an automatic metric like COMET for a sanity check, and then have a human post-edit. Yes, it costs more, but it’s worth it. For low-stakes, internal content, you can get away with raw MT plus a quick spell-check. But for customer-facing material, invest in quality. Remember, the EU’s DGT produced 2.6 million translated pages in 2022 (European Commission translation department)—they could use MT for everything, but they don’t. Follow their lead.

Sources

• Machine translation (Wikipedia) - https://en.wikipedia.org/wiki/Machine_translation
• IBM Research - https://research.ibm.com/blog/bleu-nlp-benchmark-anniversary
• COMET (ACL Anthology) - https://aclanthology.org/2020.emnlp-main.213/
• chrF (ACL Anthology) - https://aclanthology.org/W15-3049/
• Translation Edit Rate (ACL Anthology) - https://aclanthology.org/2006.amta-papers.25/
• European Commission translation department - https://commission.europa.eu/about-european-commission/departments-and-executive-agencies/translation_en