Self-training Improves Pre-training for Natural Language Understanding

Abstract

Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.

8 pages

References in corpus (15)

• Distilling the Knowledge in a Neural Network
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• Unsupervised Data Augmentation for Consistency Training
• Rethinking Pre-training and Self-training
• Billion-scale semi-supervised learning for image classification
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• Improved Noisy Student Training for Automatic Speech Recognition
• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Revisiting Self-Training for Neural Sequence Generation
• Understanding Back-Translation at Scale
• Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Simple and Effective Paraphrastic Similarity from Parallel Translations

Cited by in corpus (21)

• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Few-Shot Named Entity Recognition: A Comprehensive Study
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• Self-training and Pre-training are Complementary for Speech Recognition
• XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation
• Multi-Task Self-Training for Learning General Representations
• Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation
• Streaming Self-Training via Domain-Agnostic Unlabeled Images
• Improving Cross-Lingual Reading Comprehension with Self-Training
• CoRI: Collective Relation Integration with Data Augmentation for Open Information Extraction
• LOGEN: Few-shot Logical Knowledge-Conditioned Text Generation with Self-training
• Nearest Neighbour Few-Shot Learning for Cross-lingual Classification
• Pretext Tasks selection for multitask self-supervised speech representation learning
• Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP
• Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Data
• Predictions For Pre-training Language Models
• Adding more data does not always help: A study in medical conversation summarization with PEGASUS
• Revisiting Self-Training for Few-Shot Learning of Language Model
• Knowing More About Questions Can Help: Improving Calibration in Question Answering
• Credal Self-Supervised Learning