Generative Data Augmentation for Commonsense Reasoning

Abstract

Recent advances in commonsense reasoning depend on large-scale human-annotated training data to achieve peak performance. However, manual curation of training examples is expensive and has been shown to introduce annotation artifacts that neural models can readily exploit and overfit on. We investigate G-DAUG^C, a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting. Our approach generates synthetic examples using pretrained language models, and selects the most informative and diverse set of examples for data augmentation. In experiments with multiple commonsense reasoning benchmarks, G-DAUG^C consistently outperforms existing data augmentation methods based on back-translation, and establishes a new state-of-the-art on WinoGrande, CODAH, and CommonsenseQA. Further, in addition to improvements in in-distribution accuracy, G-DAUG^C-augmented training also enhances out-of-distribution generalization, showing greater robustness against adversarial or perturbed examples. Our analysis demonstrates that G-DAUG^C produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance. Our findings encourage future research toward generative data augmentation to enhance both in-distribution learning and out-of-distribution generalization.

arXiv commentFindings of the Association for Computational Linguistics: EMNLP 2020

References in corpus (17)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Character-level Convolutional Networks for Text Classification
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Unsupervised Data Augmentation for Consistency Training
• Understanding Black-box Predictions via Influence Functions
• The Curious Case of Neural Text Degeneration
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Generating Natural Adversarial Examples
• Synthetic and Natural Noise Both Break Neural Machine Translation
• A Simple Method for Commonsense Reasoning
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Adversarial Filters of Dataset Biases
• Data Augmentation using Pre-trained Transformer Models
• Recent Advances in Neural Question Generation
• Not Enough Data? Deep Learning to the Rescue!
• PyHessian: Neural Networks Through the Lens of the Hessian

Cited by in corpus (8)

• A Survey on Data Augmentation for Text Classification
• True Few-Shot Learning with Language Models
• HPC-GPT: Integrating Large Language Model for High-Performance Computing
• A Survey on Recent Advances in Conversational Data Generation
• Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
• On Sample Based Explanation Methods for NLP:Efficiency, Faithfulness, and Semantic Evaluation
• STraTA: Self-Training with Task Augmentation for Better Few-shot Learning