Entailment as Few-Shot Learner

Abstract

Large pre-trained language models (LMs) have demonstrated remarkable ability as few-shot learners. However, their success hinges largely on scaling model parameters to a degree that makes it challenging to train and serve. In this paper, we propose a new approach, named as EFL, that can turn small LMs into better few-shot learners. The key idea of this approach is to reformulate potential NLP task into an entailment one, and then fine-tune the model with as little as 8 examples. We further demonstrate our proposed method can be: (i) naturally combined with an unsupervised contrastive learning-based data augmentation method; (ii) easily extended to multilingual few-shot learning. A systematic evaluation on 18 standard NLP tasks demonstrates that this approach improves the various existing SOTA few-shot learning methods by 12\%, and yields competitive few-shot performance with 500 times larger models, such as GPT-3.

References in corpus (15)

• Character-level Convolutional Networks for Text Classification
• Language Models are Few-Shot Learners
• Cross-lingual Language Model Pretraining
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• A large annotated corpus for learning natural language inference
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Revisiting Few-sample BERT Fine-tuning
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• XNLI: Evaluating Cross-lingual Sentence Representations
• Making Pre-trained Language Models Better Few-shot Learners
• Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Zero-shot Text Classification With Generative Language Models
• Commonsense Knowledge Mining from Pretrained Models
• Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start

Cited by in corpus (20)

• ChatGPT: Jack of all trades, master of none
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• True Few-Shot Learning with Language Models
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• Paradigm Shift in Natural Language Processing
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
• Mask-guided BERT for Few Shot Text Classification
• NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models
• Primer: Searching for Efficient Transformers for Language Modeling
• Explainable Natural Language Processing with Matrix Product States
• Sentiment Analysis Using Averaged Weighted Word Vector Features
• Plug-Tagger: A Pluggable Sequence Labeling Framework Using Language Models
• CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
• Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction
• Open Aspect Target Sentiment Classification with Natural Language Prompts
• Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpus
• STraTA: Self-Training with Task Augmentation for Better Few-shot Learning