Testing Machine Translation via Referential Transparency

Abstract

Machine translation software has seen rapid progress in recent years due to the advancement of deep neural networks. People routinely use machine translation software in their daily lives, such as ordering food in a foreign restaurant, receiving medical diagnosis and treatment from foreign doctors, and reading international political news online. However, due to the complexity and intractability of the underlying neural networks, modern machine translation software is still far from robust and can produce poor or incorrect translations; this can lead to misunderstanding, financial loss, threats to personal safety and health, and political conflicts. To address this problem, we introduce referentially transparent inputs (RTIs), a simple, widely applicable methodology for validating machine translation software. A referentially transparent input is a piece of text that should have similar translations when used in different contexts. Our practical implementation, Purity, detects when this property is broken by a translation. To evaluate RTI, we use Purity to test Google Translate and Bing Microsoft Translator with 200 unlabeled sentences, which detected 123 and 142 erroneous translations with high precision (79.3% and 78.3%). The translation errors are diverse, including examples of under-translation, over-translation, word/phrase mistranslation, incorrect modification, and unclear logic.

arXiv commentAccepted by ICSE21

References in corpus (19)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Convolutional Sequence to Sequence Learning
• Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
• DeepXplore: Automated Whitebox Testing of Deep Learning Systems
• Achieving Human Parity on Automatic Chinese to English News Translation
• Robust Physical-World Attacks on Deep Learning Models
• Metamorphic Testing: A New Approach for Generating Next Test Cases
• Guiding Deep Learning System Testing using Surprise Adequacy
• Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition
• On Adversarial Examples for Character-Level Neural Machine Translation
• Machine Learning Testing: Survey, Landscapes and Horizons
• DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing
• Metric Learning for Adversarial Robustness
• An Empirical Comparison of Simple Domain Adaptation Methods for Neural Machine Translation
• Concolic Testing for Deep Neural Networks
• SirenAttack: Generating Adversarial Audio for End-to-End Acoustic Systems
• Testing Untestable Neural Machine Translation: An Industrial Case
• DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks