Stop Post-Editing Blind: Build a Localization QA Loop That Actually Works

Who This Is For

You're a localization manager, a lead translator, or a solo operator who's past the honeymoon phase with machine translation. You've got a CAT tool, maybe a translation memory, and you're using MT for a chunk of your content. But you're not sure if you're post-editing efficiently, or if the MT is actually getting better. This piece is for you. I'm going to walk you through a practical QA loop that treats post-editing as a feedback system, not a one-off cleanup. And I'm going to make a contrarian claim: the biggest bottleneck in localization today isn't the MT engine—it's your post-editing workflow.

Most teams I see treat post-editing as a black box: send text to MT, get draft, edit it, ship it. They never close the loop. They never measure whether the MT is improving, nor do they send corrections back to the engine. That's a missed opportunity. The tools to fix this exist, and they're not new algorithms—they're metrics and processes you probably already have access to.

Step 1: Define What “Good” Means—and Stop Relying on BLEU

First, you need a definition of quality that aligns with your actual end use. For years, we leaned on BLEU, the metric introduced by IBM researchers in 2002 (IBM Research). BLEU compares n-gram matches between machine output and human reference translations, scoring on a 0 to 1 scale. It's cheap and easy, but it's increasingly seen as unreliable because it correlates poorly with human judgment (IBM Research). So if you're using BLEU as your sole quality gate, you're fooling yourself.

Instead, adopt a human-in-the-loop metric. For post-editing effort, HTER (Human-targeted Translation Edit Rate) is a widely used proxy (Translation Edit Rate (ACL Anthology)). You create a reference that a human edits from the MT output, then measure the edit distance. That tells you how much work your post-editors are doing. But even HTER doesn't capture nuance. You also want a metric like COMET, a neural framework that has been shown to correlate better with human judgment (COMET (ACL Anthology)). Or BLEURT, a BERT-based metric from Google Research that also outperforms BLEU (BLEURT (Google Research)). The point: pick a metric that reflects human effort, not just n-gram overlap.

Step 2: Build a Feedback Loop from Post-Editors to MT

Now, the core of the loop: your post-editors are your best source of training data. When they edit MT output, they're implicitly telling you what the MT got wrong. Capture that. In your CAT tool, make sure you're tracking the original MT segment and the final human-edited segment. That pair is gold. It can be used to retrain your MT engine, or at least to build a better translation memory.

Here's a concrete example: say you're localizing a SaaS product into German. Your MT consistently mistranslates “log in” as “einloggen” but your brand style guide says “anmelden.” Your post-editor fixes it every time. If you're not capturing that correction, the next time the MT sees “log in,” it'll make the same mistake. But if you feed that correction back, either into your TM or into an MT tuning process, the engine learns. This is exactly how the European Commission's eTranslation service works—it's a neural MT service that is continuously improved by feedback from its users (eTranslation (European Commission)). You can do the same, even on a smaller scale.

Step 3: Use Translation Memory to Reinforce Consistency

Translation memory (TM) is your friend here. A TM is a database of previously translated segments that can be reused, ensuring consistency and reducing repeated work (Machine translation (Wikipedia)). When you post-edit MT, the edited segment goes into your TM. Over time, your TM becomes a storehouse of high-quality, human-approved translations. Then, when a similar sentence comes up, your CAT tool can pre-fill it from the TM, and you skip MT altogether. That's the ideal.

But there's a trap: if you blindly trust TM matches, you can propagate errors. So set a threshold: if a TM match is below, say, 90% fuzzy, send it to MT instead. If it's above, use the TM. That way, you're using MT only where it adds value, and you're letting your human-approved work take priority.

Step 4: Post-Edit with ISO Standards in Mind

Don't forget the standards. ISO 18587:2017 specifies requirements for full human post-editing of MT output and for post-editors' competences (ISO 18587:2017 (ISO)). That's not just bureaucratic mumbo-jumbo—it gives you a framework for what post-editing should look like. Similarly, ISO 17100:2015 covers the core processes of a translation service (ISO 17100:2015 (ISO)). Aligning your workflow with these standards ensures you have defined processes, qualified post-editors, and a quality management system. It also gives you a leg up when clients ask about your quality process.

One thing that often goes wrong: teams treat post-editing as a quick cleanup, not a professional task. They assign it to the least experienced person. That's a mistake. ISO 18587 requires specific competences for post-editors. Make sure your post-editors are trained, and treat their work as a professional service, not a mechanical chore.

Step 5: Track Metrics Over Time—and Act on Them

Finally, you need to track your quality metrics over time. Use a combination of HTER, COMET, or BLEURT, and also track your post-editing speed. If your HTER is decreasing month over month, that's a sign your MT is improving—or that your post-editors are getting faster. If it's increasing, something's wrong—maybe your content mix changed, or your MT engine got worse.

For example, let's say you're translating legal documents. You might use WIPO Translate, which is a neural MT tool specialized for patents and technical documents (WIPO Translate (WIPO)). It was first deployed for Chinese-to-English in 2016, and by 2017 expanded to 10 languages (WIPO Translate (WIPO)). If you use it, track your HTER on a sample of segments each month. If you see it creep up, you'll know to investigate—maybe the engine needs retraining, or maybe your source content has changed.

Warning: Do not let your post-editors “fix” MT output without recording what they changed. If you don't capture the edits, you lose the feedback loop, and you're back to blind post-editing.

Quick tip: Set up a weekly review of the top 10 most frequent MT errors, and send that list to your MT provider or your internal team. That simple action can drive targeted improvements.

Takeaway

Post-editing is not a one-way street. It's a feedback loop that can turn your MT from a blunt instrument into a precision tool. By defining quality with human-centric metrics, capturing post-editor corrections, leveraging translation memory, adhering to ISO standards, and tracking metrics over time, you'll build a localization QA loop that actually works. Stop post-editing blind—start closing the loop.

Sources

• Machine translation (Wikipedia) - https://en.wikipedia.org/wiki/Machine_translation
• IBM Research - https://research.ibm.com/blog/bleu-nlp-benchmark-anniversary
• Translation Edit Rate (ACL Anthology) - https://aclanthology.org/2006.amta-papers.25/
• COMET (ACL Anthology) - https://aclanthology.org/2020.emnlp-main.213/
• BLEURT (Google Research) - https://research.google/pubs/bleurt-learning-robust-metrics-for-text-generation/
• eTranslation (European Commission) - https://language-tools.ec.europa.eu/