CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding

Abstract

Data augmentation has been demonstrated as an effective strategy for improving model generalization and data efficiency. However, due to the discrete nature of natural language, designing label-preserving transformations for text data tends to be more challenging. In this paper, we propose a novel data augmentation framework dubbed CoDA, which synthesizes diverse and informative augmented examples by integrating multiple transformations organically. Moreover, a contrastive regularization objective is introduced to capture the global relationship among all the data samples. A momentum encoder along with a memory bank is further leveraged to better estimate the contrastive loss. To verify the effectiveness of the proposed framework, we apply CoDA to Transformer-based models on a wide range of natural language understanding tasks. On the GLUE benchmark, CoDA gives rise to an average improvement of 2.2% while applied to the RoBERTa-large model. More importantly, it consistently exhibits stronger results relative to several competitive data augmentation and adversarial training base-lines (including the low-resource settings). Extensive experiments show that the proposed contrastive objective can be flexibly combined with various data augmentation approaches to further boost their performance, highlighting the wide applicability of the CoDA framework.

References in corpus (21)

• A Simple Framework for Contrastive Learning of Visual Representations
• mixup: Beyond Empirical Risk Minimization
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Improved Regularization of Convolutional Neural Networks with Cutout
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Learning deep representations by mutual information estimation and maximization
• Unsupervised Data Augmentation for Consistency Training
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• AutoAugment: Learning Augmentation Policies from Data
• Semi-Supervised Learning with Ladder Networks
• Contrastive Multiview Coding
• Learning with Pseudo-Ensembles
• Wasserstein Distance Guided Representation Learning for Domain Adaptation
• Augmenting Data with Mixup for Sentence Classification: An Empirical Study
• Supervised Contrastive Learning
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Adversarial Training for Large Neural Language Models
• Learning Data Manipulation for Augmentation and Weighting
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation

Cited by in corpus (9)

• A Survey on Data Augmentation for Text Classification
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• Supporting Clustering with Contrastive Learning
• Unsupervised Document Embedding via Contrastive Augmentation
• Memory-Efficient Differentiable Transformer Architecture Search
• Attention-based Contrastive Learning for Winograd Schemas
• Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning
• Contrastive Document Representation Learning with Graph Attention Networks