GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation

Abstract

Large-scale language models such as GPT-3 are excellent few-shot learners, allowing them to be controlled via natural text prompts. Recent studies report that prompt-based direct classification eliminates the need for fine-tuning but lacks data and inference scalability. This paper proposes a novel data augmentation technique that leverages large-scale language models to generate realistic text samples from a mixture of real samples. We also propose utilizing soft-labels predicted by the language models, effectively distilling knowledge from the large-scale language models and creating textual perturbations simultaneously. We perform data augmentation experiments on diverse classification tasks and show that our method hugely outperforms existing text augmentation methods. Ablation studies and a qualitative analysis provide more insights into our approach.

Accepted to EMNLP2021 Findings; 11 pages, 7 tables, 2 figures

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• mixup: Beyond Empirical Risk Minimization
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Towards Understanding Knowledge Distillation
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detection