Soft Contextual Data Augmentation for Neural Machine Translation

Abstract

While data augmentation is an important trick to boost the accuracy of deep learning methods in computer vision tasks, its study in natural language tasks is still very limited. In this paper, we present a novel data augmentation method for neural machine translation. Different from previous augmentation methods that randomly drop, swap or replace words with other words in a sentence, we softly augment a randomly chosen word in a sentence by its contextual mixture of multiple related words. More accurately, we replace the one-hot representation of a word by a distribution (provided by a language model) over the vocabulary, i.e., replacing the embedding of this word by a weighted combination of multiple semantically similar words. Since the weights of those words depend on the contextual information of the word to be replaced, the newly generated sentences capture much richer information than previous augmentation methods. Experimental results on both small scale and large scale machine translation datasets demonstrate the superiority of our method over strong baselines.

Accepted by ACL 2019 as short paper

References in corpus (4)

• Sequence to Sequence Learning with Neural Networks
• AutoAugment: Learning Augmentation Policies from Data
• Dual Learning for Machine Translation
• On Using Monolingual Corpora in Neural Machine Translation

Cited by in corpus (18)

• A Survey on Data Augmentation for Text Classification
• Data Augmentation Approaches in Natural Language Processing: A Survey
• Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretraining
• An Analysis of Simple Data Augmentation for Named Entity Recognition
• Prevent the Language Model from being Overconfident in Neural Machine Translation
• AR: Auto-Repair the Synthetic Data for Neural Machine Translation
• Sequence-Level Mixed Sample Data Augmentation
• Investigation on Data Adaptation Techniques for Neural Named Entity Recognition
• Substructure Substitution: Structured Data Augmentation for NLP
• Distilling Knowledge from Pre-trained Language Models via Text Smoothing
• Unsupervised Document Embedding via Contrastive Augmentation
• An Empirical Study of Contextual Data Augmentation for Japanese Zero Anaphora Resolution
• Uncertainty-Aware Semantic Augmentation for Neural Machine Translation
• Data Augmentation for Cross-Domain Named Entity Recognition
• Recognising Biomedical Names: Challenges and Solutions
• Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach
• Gradient Imitation Reinforcement Learning for Low Resource Relation Extraction
• Towards Efficiently Diversifying Dialogue Generation via Embedding Augmentation