Few-shot Sequence Learning with Transformers
arXiv:2012.09543
Abstract
Few-shot algorithms aim at learning new tasks provided only a handful of training examples. In this work we investigate few-shot learning in the setting where the data points are sequences of tokens and propose an efficient learning algorithm based on Transformers. In the simplest setting, we append a token to an input sequence which represents the particular task to be undertaken, and show that the embedding of this token can be optimized on the fly given few labeled examples. Our approach does not require complicated changes to the model architecture such as adapter layers nor computing second order derivatives as is currently popular in the meta-learning and few-shot learning literature. We demonstrate our approach on a variety of tasks, and analyze the generalization properties of several model variants and baseline approaches. In particular, we show that compositional task descriptors can improve performance. Experiments show that our approach works at least as well as other methods, while being more computationally efficient.
NeurIPS Meta-Learning Workshop 2020
References in corpus (13)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Prototypical Networks for Few-shot Learning
- Language Models are Few-Shot Learners
- A Simple Neural Attentive Meta-Learner
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Learning Task Grouping and Overlap in Multi-task Learning
- Fast Context Adaptation via Meta-Learning
- Using Fast Weights to Attend to the Recent Past
- Stabilizing Transformers for Reinforcement Learning
- BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
- Mixture Models for Diverse Machine Translation: Tricks of the Trade
- Few-shot Text Classification with Distributional Signatures
- Compositional generalization through meta sequence-to-sequence learning