Structure-Invariant Testing for Machine Translation

Abstract

In recent years, machine translation software has increasingly been integrated into our daily lives. People routinely use machine translation for various applications, such as describing symptoms to a foreign doctor and reading political news in a foreign language. However, the complexity and intractability of neural machine translation (NMT) models that power modern machine translation make the robustness of these systems difficult to even assess, much less guarantee. Machine translation systems can return inferior results that lead to misunderstanding, medical misdiagnoses, threats to personal safety, or political conflicts. Despite its apparent importance, validating the robustness of machine translation systems is very difficult and has, therefore, been much under-explored. To tackle this challenge, we introduce structure-invariant testing (SIT), a novel metamorphic testing approach for validating machine translation software. Our key insight is that the translation results of "similar" source sentences should typically exhibit similar sentence structures. Specifically, SIT (1) generates similar source sentences by substituting one word in a given sentence with semantically similar, syntactically equivalent words; (2) represents sentence structure by syntax parse trees (obtained via constituency or dependency parsing); (3) reports sentence pairs whose structures differ quantitatively by more than some threshold. To evaluate SIT, we use it to test Google Translate and Bing Microsoft Translator with 200 source sentences as input, which led to 64 and 70 buggy issues with 69.5\% and 70\% top-1 accuracy, respectively. The translation errors are diverse, including under-translation, over-translation, incorrect modification, word/phrase mistranslation, and unclear logic.

arXiv commentAccepted at ICSE 2020

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Achieving Human Parity on Automatic Chinese to English News Translation
• Robust Physical-World Attacks on Deep Learning Models
• Metamorphic Testing: A New Approach for Generating Next Test Cases
• Guiding Deep Learning System Testing using Surprise Adequacy
• Document Embedding with Paragraph Vectors
• Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition
• On Adversarial Examples for Character-Level Neural Machine Translation
• Understanding Back-Translation at Scale
• Machine Learning Testing: Survey, Landscapes and Horizons
• DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing
• An Empirical Comparison of Simple Domain Adaptation Methods for Neural Machine Translation
• SirenAttack: Generating Adversarial Audio for End-to-End Acoustic Systems
• Testing Untestable Neural Machine Translation: An Industrial Case