Towards Enabling Meta-Learning from Target Models

Abstract

Meta-learning can extract an inductive bias from previous learning experience and assist the training of new tasks. It is often realized through optimizing a meta-model with the evaluation loss of task-specific solvers. Most existing algorithms sample non-overlapping $support$ sets and $query$ sets to train and evaluate the solvers respectively due to simplicity ($\mathcal{S}$/$\mathcal{Q}$ protocol). Different from $\mathcal{S}$/$\mathcal{Q}$ protocol, we can also evaluate a task-specific solver by comparing it to a target model $\mathcal{T}$, which is the optimal model for this task or a model that behaves well enough on this task ($\mathcal{S}$/$\mathcal{T}$ protocol). Although being short of research, $\mathcal{S}$/$\mathcal{T}$ protocol has unique advantages such as offering more informative supervision, but it is computationally expensive. This paper looks into this special evaluation method and takes a step towards putting it into practice. We find that with a small ratio of tasks armed with target models, classic meta-learning algorithms can be improved a lot without consuming many resources. We empirically verify the effectiveness of $\mathcal{S}$/$\mathcal{T}$ protocol in a typical application of meta-learning, $i.e.$, few-shot learning. In detail, after constructing target models by fine-tuning the pre-trained network on those hard tasks, we match the task-specific solvers and target models via knowledge distillation.

This paper has been accepted by NeurIPS'21

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• Theoretical Models of Learning to Learn
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Free Lunch for Few-shot Learning: Distribution Calibration
• Meta-Reinforcement Learning of Structured Exploration Strategies
• An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning
• Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
• Task-Robust Model-Agnostic Meta-Learning
• Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks
• Revisiting Meta-Learning as Supervised Learning
• Few-Shot Action Recognition with Compromised Metric via Optimal Transport
• Meta-Learning with Shared Amortized Variational Inference