A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation

Abstract

Adversarial training has been shown effective at endowing the learned representations with stronger generalization ability. However, it typically requires expensive computation to determine the direction of the injected perturbations. In this paper, we introduce a set of simple yet effective data augmentation strategies dubbed cutoff, where part of the information within an input sentence is erased to yield its restricted views (during the fine-tuning stage). Notably, this process relies merely on stochastic sampling and thus adds little computational overhead. A Jensen-Shannon Divergence consistency loss is further utilized to incorporate these augmented samples into the training objective in a principled manner. To verify the effectiveness of the proposed strategies, we apply cutoff to both natural language understanding and generation problems. On the GLUE benchmark, it is demonstrated that cutoff, in spite of its simplicity, performs on par or better than several competitive adversarial-based approaches. We further extend cutoff to machine translation and observe significant gains in BLEU scores (based upon the Transformer Base model). Moreover, cutoff consistently outperforms adversarial training and achieves state-of-the-art results on the IWSLT2014 German-English dataset.

arXiv commentSource code is available at: https://github.com/dinghanshen/cutoff

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Unsupervised Data Augmentation for Consistency Training
• A Survey on Multi-view Learning
• ERNIE: Enhanced Representation through Knowledge Integration
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Learning with Pseudo-Ensembles
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• An Actor-Critic Algorithm for Sequence Prediction
• Weighted Transformer Network for Machine Translation
• Adversarial Training for Large Neural Language Models
• You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
• Improving Neural Language Modeling via Adversarial Training
• Data Diversification: A Simple Strategy For Neural Machine Translation
• Multi-branch Attentive Transformer
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule

Cited by in corpus (18)

• A Survey on Data Augmentation for Text Classification
• R-Drop: Regularized Dropout for Neural Networks
• ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis
• An Empirical Survey of Data Augmentation for Limited Data Learning in NLP
• DropDim: A Regularization Method for Transformer Networks
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
• An Effective, Robust and Fairness-aware Hate Speech Detection Framework
• Unsupervised Document Embedding via Contrastive Augmentation
• Reducing and Exploiting Data Augmentation Noise through Meta Reweighting Contrastive Learning for Text Classification
• Enhancing Pre-trained Language Model with Lexical Simplification
• Rethinking Data Augmentation for Low-Resource Neural Machine Translation: A Multi-Task Learning Approach
• The NiuTrans System for the WMT21 Efficiency Task
• A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classification
• Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning