Meta-learning with differentiable closed-form solvers

Abstract

Adapting deep networks to new concepts from a few examples is challenging, due to the high computational requirements of standard fine-tuning procedures. Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent. Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efficiently. In this paper, we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning. The main idea is to teach a deep network to use standard machine learning tools, such as ridge regression, as part of its own internal model, enabling it to quickly adapt to novel data. This requires back-propagating errors through the solver steps. While normally the cost of the matrix operations involved in such a process would be significant, by using the Woodbury identity we can make the small number of examples work to our advantage. We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components. Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.

Published at ICLR'19. Code and data available at http://www.robots.ox.ac.uk/~luca/r2d2.html

References in corpus (11)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• WaveNet: A Generative Model for Raw Audio
• How transferable are features in deep neural networks?
• An Overview of Multi-Task Learning in Deep Neural Networks
• Theoretical Models of Learning to Learn
• Learning multiple visual domains with residual adapters
• Few-Shot Learning with Graph Neural Networks
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Training Deep Networks with Structured Layers by Matrix Backpropagation
• Memory-based Parameter Adaptation

Cited by in corpus (24)

• Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning
• MHFC: Multi-Head Feature Collaboration for Few-Shot Learning
• Sparse Spatial Transformers for Few-Shot Learning
• On the Iteration Complexity of Hypergradient Computation
• A Baseline for Few-Shot Image Classification
• Learning to Focus: Cascaded Feature Matching Network for Few-shot Image Recognition
• Prototype Rectification for Few-Shot Learning
• Scalable Bayesian Meta-Learning through Generalized Implicit Gradients
• Transfer learning based few-shot classification using optimal transport mapping from preprocessed latent space of backbone neural network
• ARCADe: A Rapid Continual Anomaly Detector
• Learning from the Past: Continual Meta-Learning via Bayesian Graph Modeling
• Generalized Adaptation for Few-Shot Learning
• Differentiable Meta-learning Model for Few-shot Semantic Segmentation
• Reward Optimization for Neural Machine Translation with Learned Metrics
• Meta Learning for Few-Shot One-class Classification
• How Fine-Tuning Allows for Effective Meta-Learning
• Graph Few-shot Learning via Knowledge Transfer
• SML: Semantic Meta-learning for Few-shot Semantic Segmentation
• Single-View 3D Object Reconstruction from Shape Priors in Memory
• Long-term Cross Adversarial Training: A Robust Meta-learning Method for Few-shot Classification Tasks
• Proximal Mapping for Deep Regularization
• Uniform Sampling over Episode Difficulty
• On Hard Episodes in Meta-Learning
• Self-Denoising Neural Networks for Few Shot Learning