When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classification

Abstract

Text classification tends to be difficult when data are deficient or when it is required to adapt to unseen classes. In such challenging scenarios, recent studies have often used meta-learning to simulate the few-shot task, thus negating implicit common linguistic features across tasks. This paper addresses such problems using meta-learning and unsupervised language models. Our approach is based on the insight that having a good generalization from a few examples relies on both a generic model initialization and an effective strategy for adapting this model to newly arising tasks. We show that our approach is not only simple but also produces a state-of-the-art performance on a well-studied sentiment classification dataset. It can thus be further suggested that pretraining could be a promising solution for few-shot learning of many other NLP tasks. The code and the dataset to replicate the experiments are made available at https://github.com/zxlzr/FewShotNLP.

arXiv commentAAAI student abstract

References in corpus (8)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• A Structured Self-attentive Sentence Embedding
• Unsupervised Domain Adaptation with Residual Transfer Networks
• Matching Networks for One Shot Learning
• Compositional generalization in a deep seq2seq model by separating syntax and semantics
• Induction Networks for Few-Shot Text Classification
• Attention Focusing for Neural Machine Translation by Bridging Source and Target Embeddings

Cited by in corpus (5)

• Conceptualized Representation Learning for Chinese Biomedical Text Mining
• CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detection
• An Empirical Survey of Data Augmentation for Limited Data Learning in NLP
• Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference
• Knowledge Guided Metric Learning for Few-Shot Text Classification