Empirical Bayes Transductive Meta-Learning with Synthetic Gradients

Abstract

We propose a meta-learning approach that learns from multiple tasks in a transductive setting, by leveraging the unlabeled query set in addition to the support set to generate a more powerful model for each task. To develop our framework, we revisit the empirical Bayes formulation for multi-task learning. The evidence lower bound of the marginal log-likelihood of empirical Bayes decomposes as a sum of local KL divergences between the variational posterior and the true posterior on the query set of each task. We derive a novel amortized variational inference that couples all the variational posteriors via a meta-model, which consists of a synthetic gradient network and an initialization network. Each variational posterior is derived from synthetic gradient descent to approximate the true posterior on the query set, although where we do not have access to the true gradient. Our results on the Mini-ImageNet and CIFAR-FS benchmarks for episodic few-shot classification outperform previous state-of-the-art methods. Besides, we conduct two zero-shot learning experiments to further explore the potential of the synthetic gradient.

arXiv commentICLR 2020

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• The information bottleneck method
• Wide Residual Networks
• Theoretical Models of Learning to Learn
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• On First-Order Meta-Learning Algorithms
• A simple neural network module for relational reasoning
• A Closer Look at Few-shot Classification
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• Self-supervised Domain Adaptation for Computer Vision Tasks
• Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theory
• Transferring Knowledge across Learning Processes
• Population Empirical Bayes

Cited by in corpus (9)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• MHFC: Multi-Head Feature Collaboration for Few-Shot Learning
• Meta-Learned Confidence for Few-shot Learning
• Graph-based Interpolation of Feature Vectors for Accurate Few-Shot Classification
• Transductive Few-Shot Learning: Clustering is All You Need?
• Learning by Examples Based on Multi-level Optimization
• Meta-Learning with Task-Adaptive Loss Function for Few-Shot Learning
• Adaptive Pseudo-Label Refinement by Negative Ensemble Learning for Source-Free Unsupervised Domain Adaptation
• Few-shot Learning with LSSVM Base Learner and Transductive Modules