CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

Abstract

Humans can learn a new language task efficiently with only few examples, by leveraging their knowledge obtained when learning prior tasks. In this paper, we explore whether and how such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a problem setup for studying cross-task generalization ability, which standardizes seen/unseen task partitions, data access during different learning stages, and the evaluation protocols. To instantiate different seen/unseen task partitions in CrossFit and facilitate in-depth analysis, we present the NLP Few-shot Gym, a repository of 160 diverse few-shot NLP tasks created from open-access NLP datasets and converted to a unified text-to-text format. Our analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. We also observe that the selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.

arXiv commentAccepted to EMNLP 2021. Camera-ready version. Code: https://github.com/INK-USC/CrossFit

References in corpus (16)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Wizard of Wikipedia: Knowledge-Powered Conversational agents
• On First-Order Meta-Learning Algorithms
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization
• The Natural Language Decathlon: Multitask Learning as Question Answering
• TabFact: A Large-scale Dataset for Table-based Fact Verification
• Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challenge
• Entailment as Few-Shot Learner
• How Context Affects Language Models' Factual Predictions
• Finetuned Language Models Are Zero-Shot Learners
• Understanding and Improving Information Transfer in Multi-Task Learning
• CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims

Cited by in corpus (1)

• Finetuned Language Models Are Zero-Shot Learners