Should You Trust Machine Translation for Your Product? A Localization Editor's Take

You've got a product ready for the global market, and someone says, "Just run it through Google Translate or DeepL. It's good enough now." Is it? As an editor who's watched this field evolve from the statistical dark ages to the neural revolution, I have a strong opinion: machine translation is a fantastic starting point, but it is not a finished product. The question isn't whether to use MT—it's how to use it responsibly.

The Allure and the Trap of Modern NMT

I remember when Google announced its neural machine translation system in 2016. The claim was that it reduced translation errors by 55–85% on several major language pairs compared to the old phrase-based system (GNMT at production scale). That was a huge leap. DeepL came along in 2017 and reportedly won blind tests with professional translators about three times more often than Google, Microsoft, or Facebook (DeepL Translator). It's tempting to think the machines have us covered.

But here's the trap: NMT is trained on vast amounts of text, but it doesn't understand context the way a human does. It can produce fluent-sounding nonsense. For a product, a single mistranslated button label or a confusing error message can make your tool look unprofessional. The technology is impressive, but it's not a substitute for human judgment.

When MT Can Shine (and When It Can't)

Let's be practical. If you're translating a user manual for a generic product, MT might get you 80% of the way there. If you're localizing a marketing site with puns, cultural references, or emotional appeals, MT will likely fall flat. The fact base notes that MT often requires human post-editing, especially for idiomatic expressions, cultural nuance, and domain-specific terminology (Machine translation). That's the key: know the nature of your content.

For high-stakes content like legal disclaimers, medical instructions, or anything where a mistake could be costly, never go live without a human review. For low-stakes, high-volume content like support articles, MT with light post-editing can be a lifesaver. The European Commission's Directorate-General for Translation, one of the world's largest, still produced 2.6 million translated pages in 2022 (European Commission translation department)—they use eTranslation, their free neural MT service, but they don't just publish raw output. They have a process.

Why BLEU Is Lying to You

For years, we've been told to trust BLEU scores. BLEU was introduced in 2002 and compares n-gram matches between machine output and human references (IBM Research). It's a quick, cheap way to benchmark systems, but it correlates poorly with human judgment. I've seen systems score high BLEU yet produce awkward, unnatural text. The fact base is clear: BLEU is increasingly seen as unreliable, and newer metrics like COMET and BLEURT are becoming standard (IBM Research).

COMET, developed by Unbabel, is a neural framework that achieved top performance at WMT 2019 and 2020 (COMET). BLEURT, from Google, correlates better with human judgment on several tasks (BLEURT). If you're serious about MT quality, use these human-aligned metrics, not BLEU. And remember, the ultimate metric is post-editing effort: how much time a human needs to fix the output. That's what Translation Edit Rate (TER) tries to capture, and its human-targeted variant (HTER) is a good proxy for that effort (Translation Edit Rate).

Post-Editing: The Human in the Loop

Here's my strong recommendation: embrace post-editing. ISO 18587:2017 sets standards for full human post-editing of MT output, specifying the competences required (ISO 18587:2017). This isn't a dirty secret; it's a professional practice. The key is to define the level of quality you need. For gist translation, light post-editing might suffice. For publication, you need full post-editing, where a human ensures fluency, accuracy, and cultural appropriateness.

Let me give you a concrete example. Say you're localizing a fitness app into Spanish. The phrase "don't overtrain" might be translated by MT as "no sobreentrenes," which is fine, but a human post-editor might prefer "evita el sobreentrenamiento" for a more natural tone. That's the difference between acceptable and polished. And for a product, polished matters.

Tools to Help You Localize Smarter

You don't have to do this manually. Computer-assisted translation (CAT) tools are designed to support human translators, not replace them (Machine translation). They use translation memory (TM) to store previously translated segments, ensuring consistency and reducing repeated work. XLIFF is the standard format for exchanging these files, and it lets you define translation units—a word, sentence, or paragraph—with larger units generally giving more idiomatic translations (Localization file formats).

So, the workflow I recommend is this: use MT to get a draft, then use a CAT tool with your TM and a human post-editor to refine it. This is exactly what Google Translator Toolkit did before it shut down—it pre-translated from TM and fell back on MT (Google Translator Toolkit). The technology has moved on, but the principle holds.

Quick Tip

Before you commit to MT for a new language, run a small pilot. Translate a few pages, have a professional post-edit them, and measure the actual time and cost. That data will tell you more than any benchmark.

Bottom Line

The single best move you can make is to treat machine translation as an assistant, not a replacement. Use NMT for a first pass, but invest in post-editing by a qualified human—and measure quality with modern metrics like COMET or BLEURT, not BLEU. That's how you get speed and quality without the risk.

Sources

• Machine translation (Wikipedia) - https://en.wikipedia.org/wiki/Machine_translation
• IBM Research - https://research.ibm.com/blog/bleu-nlp-benchmark-anniversary
• DeepL Translator (Wikipedia) - https://en.wikipedia.org/wiki/DeepL_Translator
• ISO 18587:2017 (ISO) - https://www.iso.org/standard/62970.html
• COMET (ACL Anthology) - https://aclanthology.org/2020.emnlp-main.213/
• GNMT at production scale (Google Research) - https://research.google/blog/a-neural-network-for-machine-translation-at-production-scale/