How Should Translation Theory Guide Post-Editing Tool Choice?

What’s the best way to measure machine translation quality when you’re a working translator?

You’ve just run a batch of technical documents through DeepL or Google Translate, and you need to know which segments are worth post-editing and which are better off redone from scratch. The standard answer in the industry used to be BLEU, but after years of relying on it, we’ve learned that BLEU mostly measures how close a machine output is to one or two human references—not how much effort you’ll actually spend fixing it. So, what should you use? In this head-to-head, we compare three evaluation metrics—BLEU, TER, and COMET—on four criteria that matter in a real workflow: correlation with human judgment, sensitivity to meaning changes, practical usability, and alignment with post-editing effort.

The Contenders: BLEU, TER, and COMET

Let’s start with the old guard. BLEU (Bilingual Evaluation Understudy) was introduced by IBM researchers in 2002 and quickly became the default metric for machine translation evaluation. It compares n-grams (sequences of words) between the machine output and a reference translation, giving a score from 0 to 1 (or 0 to 100) based on precision. The problem, as IBM Research itself now acknowledges, is that BLEU correlates poorly with human judgment—especially for sentences where a meaning-preserving synonym or a word-order change lowers the score even though a human would accept it. For a translator, that’s a red flag: you might discard a perfectly usable MT segment because BLEU says it’s bad, or keep one that BLEU says is good but that actually contains a subtle mistranslation.

Next, TER (Translation Edit Rate) takes a different approach. Instead of counting n-gram overlaps, TER measures how many edits (insertions, deletions, substitutions, and shifts) are needed to change the machine output into a reference translation. That sounds more aligned with post-editing effort, but it still depends on a single reference—if the reference isn’t the one you would have written, the edit count can be misleading. A variant called HTER (Human-targeted TER) has a human create a reference that is as close as possible to the machine output, and that variant is widely used as a proxy for post-editing effort (Translation Edit Rate, ACL Anthology). For a working translator, HTER is far more informative than plain TER because it measures the minimum edits you’d actually make.

Finally, COMET is a neural framework that was developed by Unbabel and achieved top performance at the WMT 2019 and WMT 2020 metrics shared tasks (COMET, ACL Anthology). Instead of counting surface matches, COMET uses a neural model to predict how a human would rate the translation—taking into account meaning, fluency, and adequacy. It’s the kind of metric that can catch paraphrases that BLEU misses, and it correlates better with human judgment. That makes it a strong candidate for guiding post-editing decisions.

Comparing on Four Criteria

Let’s lay out the comparison. The table below summarizes how BLEU, TER, and COMET stack up on the criteria that matter most when you’re deciding whether to post-edit or retranslate.

What does this mean for you? BLEU is still useful for head-to-head comparisons of systems in a research setting, but it’s unreliable for judging a single output. For example, the Transformer architecture improved BLEU by more than 2 points on WMT 2014 English-to-German (Attention Is All You Need, NeurIPS 2017), which is meaningful for system development, but a 2-point difference in BLEU doesn’t tell you how many minutes you’ll spend fixing a translation.

Who Should Use Which Metric?

If you’re a translation agency evaluating multiple MT engines (DeepL, Google, Microsoft, Amazon) to decide which one to deploy for a client, BLEU might be a quick way to filter candidates, but you should not rely on it alone. The same applies if you’re a researcher developing a new MT model—BLEU is still the baseline, but you should supplement it with COMET to get closer to human judgment. For a freelance translator or a small team post-editing MT output, the most practical approach is to use HTER: create a human reference that is as close as possible to the machine output, then measure the edit distance. That directly mirrors the effort you’ll spend, and it’s what ISO 18587:2017 implicitly asks for when it requires full human post-editing (ISO 18587:2017, ISO). In fact, many translation memory tools now integrate MT and can track edit distance, which is essentially HTER in practice.

On the other hand, if you need a metric that can be automated and that correlates well with human quality judgments, COMET is the best choice. It’s particularly good at detecting meaning shifts that BLEU misses. But COMET requires a reference translation, which you don’t always have. If you’re working with a document that has no previous human translation, you can’t use COMET directly. In that case, you might use quality estimation tools that predict COMET-like scores without a reference, but that’s a different topic.

Quick tip: Don’t use BLEU to decide whether to post-edit a single sentence. Use HTER if you can, or at least read the output critically. BLEU is a blunt instrument.

What I’d Actually Do

Here’s my recommendation, based on years of working with MT and CAT tools: For any serious workflow, adopt COMET as your primary automated metric for system selection and continuous monitoring. When you’re actually post-editing, switch to HTER—it’s the only metric that directly reflects your effort. And when you’re evaluating a new MT engine, don’t trust BLEU alone; run it through COMET and also have a human spot-check. That may sound like extra work, but it saves you from the false confidence that BLEU gives you. In a real scenario, say you’re translating a 10,000-word technical manual from English to German. You run it through Google Translate and get a BLEU score of 45, which sounds decent. But when you start post-editing, you find that many sentences are grammatically correct but mistranslate technical terms—a problem BLEU doesn’t catch. If you had used COMET, you might have spotted the low adequacy scores and decided to translate those sections from scratch, saving hours. Now, with MT quality improving—Google’s GNMT reduced errors by 55–85% compared to phrase-based systems on sampled sentences (GNMT at production scale, Google Research)—the gap between metrics matters even more. So my advice: invest in learning how to use COMET and HTER. They’re not perfect, but they’re light-years ahead of BLEU for practical decision-making.

Sources

• IBM Research - https://research.ibm.com/blog/bleu-nlp-benchmark-anniversary
• COMET (ACL Anthology) - https://aclanthology.org/2020.emnlp-main.213/
• Translation Edit Rate (ACL Anthology) - https://aclanthology.org/2006.amta-papers.25/
• GNMT at production scale (Google Research) - https://research.google/blog/a-neural-network-for-machine-translation-at-production-scale/
• Attention Is All You Need (NeurIPS 2017) - https://arxiv.org/abs/1706.03762
• ISO 18587:2017 (ISO) - https://www.iso.org/standard/62970.html