Towards Zero-Label Language Learning

Abstract

This paper explores zero-label learning in Natural Language Processing (NLP), whereby no human-annotated data is used anywhere during training and models are trained purely on synthetic data. At the core of our framework is a novel approach for better leveraging the powerful pretrained language models. Specifically, inspired by the recent success of few-shot inference on GPT-3, we present a training data creation procedure named Unsupervised Data Generation (UDG), which leverages few-shot prompts to synthesize high-quality training data without real human annotations. Our method enables zero-label learning as we train task-specific models solely on the synthetic data, yet we achieve better or comparable results from strong baseline models trained on human-labeled data. Furthermore, when mixed with labeled data, our approach serves as a highly effective data augmentation procedure, achieving new state-of-the-art results on the SuperGLUE benchmark.

References in corpus (12)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Character-level Convolutional Networks for Text Classification
• Language Models are Few-Shot Learners
• Unsupervised Data Augmentation for Consistency Training
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Towards a Human-like Open-Domain Chatbot
• Data Augmentation using Pre-trained Transformer Models
• Language Models as Knowledge Bases?
• Language Models are Open Knowledge Graphs
• Neural Data Augmentation via Example Extrapolation
• Not Enough Data? Deep Learning to the Rescue!
• DARE: Data Augmented Relation Extraction with GPT-2

Cited by in corpus (3)

• Finetuned Language Models Are Zero-Shot Learners
• Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialogue
• Automated Annotation of Scientific Texts for ML-based Keyphrase Extraction and Validation