SemMT: A Semantic-based Testing Approach for Machine Translation Systems

Abstract

Machine translation has wide applications in daily life. In mission-critical applications such as translating official documents, incorrect translation can have unpleasant or sometimes catastrophic consequences. This motivates recent research on testing methodologies for machine translation systems. Existing methodologies mostly rely on metamorphic relations designed at the textual level (e.g., Levenshtein distance) or syntactic level (e.g., the distance between grammar structures) to determine the correctness of translation results. However, these metamorphic relations do not consider whether the original and translated sentences have the same meaning (i.e., Semantic similarity). Therefore, in this paper, we propose SemMT, an automatic testing approach for machine translation systems based on semantic similarity checking. SemMT applies round-trip translation and measures the semantic similarity between the original and translated sentences. Our insight is that the semantics expressed by the logic and numeric constraint in sentences can be captured using regular expressions (or deterministic finite automata) where efficient equivalence/similarity checking algorithms are available. Leveraging the insight, we propose three semantic similarity metrics and implement them in SemMT. The experiment result reveals SemMT can achieve higher effectiveness compared with state-of-the-art works, achieving an increase of 21% and 23% on accuracy and F-Score, respectively. We also explore potential improvements that can be achieved when proper combinations of metrics are adopted. Finally, we discuss a solution to locate the suspicious trip in round-trip translation, which may shed lights on further exploration.

References in corpus (4)

• Explaining and Harnessing Adversarial Examples
• SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
• Metamorphic Testing: A New Approach for Generating Next Test Cases
• Revisiting Round-Trip Translation for Quality Estimation

Cited by in corpus (7)

• MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases
• RetAssist: Facilitating Vocabulary Learners with Generative Images in Story Retelling Practices
• A Survey of the Metrics, Uses, and Subjects of Diversity-Based Techniques in Software Testing
• MR-Adopt: Automatic Deduction of Input Transformation Function for Metamorphic Testing
• Word Closure-Based Metamorphic Testing for Machine Translation
• ROME: Testing Image Captioning Systems via Recursive Object Melting
• Lost In Translation: Generating Adversarial Examples Robust to Round-Trip Translation