Revisiting Self-Training for Few-Shot Learning of Language Model

Abstract

As unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model. The question is how to effectively make use of such data. In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM. Given two views of a text sample via weak and strong augmentation techniques, SFLM generates a pseudo label on the weakly augmented version. Then, the model predicts the same pseudo label when fine-tuned with the strongly augmented version. This simple approach is shown to outperform other state-of-the-art supervised and semi-supervised counterparts on six sentence classification and six sentence-pair classification benchmarking tasks. In addition, SFLM only relies on a few in-domain unlabeled data. We conduct a comprehensive analysis to demonstrate the robustness of our proposed approach under various settings, including augmentation techniques, model scale, and few-shot knowledge transfer across tasks.

arXiv commentAccepted to EMNLP 2021

References in corpus (8)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• AutoAugment: Learning Augmentation Policies from Data
• Rethinking Pre-training and Self-training
• Few-shot Text Classification with Distributional Signatures
• Self-training Improves Pre-training for Natural Language Understanding