Why You Should Stop Trusting BLEU and Start Post-Editing Like a Pro

You’ve been lied to. Not by a person, but by a metric. BLEU, the score that supposedly tells you whether a machine translation is good, has been propping up bad decisions for over two decades. It’s time you stopped treating it as gospel. Here’s the blunt truth: BLEU is a crutch, and you’re better off without it.

The question this article answers is simple: How should you evaluate machine translation output when the stakes are high? Not by staring at a single number, but by combining a modern neural metric with a human post-editing pass. That’s the only way to actually deliver quality in a world where MT is everywhere—and where your reputation is on the line.

The BLEU Delusion

BLEU was born in 2002 at IBM, a clever idea to score translation quality by counting n-gram overlaps between machine output and human references. On the surface, it made sense: the more words match, the better. But here’s the catch—BLEU correlates poorly with human judgment. IBM itself now admits it’s increasingly unreliable (IBM Research). Yet, because it’s cheap and fast, it became the default in MT research and in many translation workflows.

Why does this matter to you? Because if you’re using BLEU to decide whether a translation is good enough to ship, you’re making decisions based on a metric that can be gamed. A translation can have high BLEU and still read like garbage. Or it can have low BLEU and be perfectly acceptable. BLEU doesn’t know idiom, doesn’t know context, and doesn’t know your brand’s voice. It just counts matching words.

So, what do you do instead? You stop relying on a single score and start using tools that actually predict human quality—and you pair them with a human in the loop.

Modern Metrics That Actually Work

Since BLEU’s heyday, researchers have built better metrics. METEOR, from Carnegie Mellon, improved on BLEU by using both precision and recall and matching synonyms and paraphrases—so it’s less brittle. chrF, proposed in 2015, uses character n-grams, making it robust across languages and tokenizations, especially for morphologically rich tongues. But the real leap came with neural metrics like COMET and BLEURT. COMET, developed by Unbabel, topped the WMT metrics shared task in 2019 and 2020 (ACL Anthology). BLEURT, from Google, is BERT-based and correlates better with human judgment than BLEU on several generation tasks (Google Research). These aren’t just academic toys—they’re practical tools you can run on your own translations to get a quality signal that’s far closer to how a human would score it.

Here’s a comparison to make it concrete:

If you’re in a production environment, skip BLEU entirely. Use COMET or BLEURT for a quick check, but never as the final say.

The Post-Editing Imperative

Here’s the part that hurts: no metric, however smart, replaces a human. The ISO 18587:2017 standard exists precisely because MT output often needs human post-editing—especially for idiomatic expressions, cultural nuance, and domain-specific terminology (ISO). When you post-edit, you’re not just fixing typos; you’re ensuring the translation actually communicates. And there’s a metric for that too: Human-targeted Translation Edit Rate (HTER), a variant of TER where a human creates a reference close to the MT output, is widely used as a proxy for post-editing effort (ACL Anthology).

Consider a real scenario: you’re translating a legal contract from English to Japanese. A neural MT system might produce a grammatically correct translation, but a single mistranslated clause can have legal consequences. A BLEU score of 0.7 doesn’t tell you that. A human post-editor, following ISO 18587, will catch the nuance and fix it. That’s why the European Commission’s Directorate-General for Translation, one of the largest translation services in the world, produces 2.6 million pages a year—and still relies on human translators (European Commission). They don’t ship raw MT.

So, here’s my blunt advice: never use MT output without a post-editing pass, unless you’re translating something trivial and disposable. For anything that matters—marketing copy, medical instructions, legal documents—you need a human in the loop.

Practical Steps for Your Workflow

First, if you’re using a CAT tool, you’re already ahead—CAT tools aid human translators, not replace them, and they use translation memory to boost consistency (Machine translation (Wikipedia)). But don’t let the tool’s built-in MT suggestions go unchecked. Configure it to use a modern neural MT engine (like DeepL or Google, which both use NMT) but always review the output.

Second, when you evaluate MT quality, use COMET or BLEURT instead of BLEU. Run them on a sample of your translations to get a sense of where the system struggles. If you see low scores, allocate more time for post-editing.

Third, measure post-editing effort with HTER. Track how many edits your post-editors make. This gives you a data-driven way to decide whether a particular MT engine is worth using for a given language pair or domain.

Fourth, and this is crucial, implement a quality gate. Before you deliver, have a second human review the post-edited output, especially for high-risk projects. ISO 17100:2015 sets requirements for translation services, and ISO 18587 covers post-editing—use them as your checklist.

Quick tip: If you’re working with a language pair where MT is weak—like some African or Indian languages—expect to need full post-editing, not just light touch-ups. Meta’s NLLB-200 model improved quality by about 44% on average, but even that isn’t enough to skip humans (Nature).

Sources

• IBM Research - https://research.ibm.com/blog/bleu-nlp-benchmark-anniversary
• COMET (ACL Anthology) - https://aclanthology.org/2020.emnlp-main.213/
• BLEURT (Google Research) - https://research.google/pubs/bleurt-learning-robust-metrics-for-text-generation/
• ISO 18587:2017 - https://www.iso.org/standard/62970.html
• Translation Edit Rate (ACL Anthology) - https://aclanthology.org/2006.amta-papers.25/
• European Commission translation department - https://commission.europa.eu/about-european-commission/departments-and-executive-agencies/translation_en