Covariate Distribution Aware Meta-learning

Abstract

Meta-learning has proven to be successful for few-shot learning across the regression, classification, and reinforcement learning paradigms. Recent approaches have adopted Bayesian interpretations to improve gradient-based meta-learners by quantifying the uncertainty of the post-adaptation estimates. Most of these works almost completely ignore the latent relationship between the covariate distribution $(p(x))$ of a task and the corresponding conditional distribution $p(y|x)$. In this paper, we identify the need to explicitly model the meta-distribution over the task covariates in a hierarchical Bayesian framework. We begin by introducing a graphical model that leverages the samples from the marginal $p(x)$ to better infer the posterior over the optimal parameters of the conditional distribution $(p(y|x))$ for each task. Based on this model we propose a computationally feasible meta-learning algorithm by introducing meaningful relaxations in our final objective. We demonstrate the gains of our algorithm over initialization based meta-learning baselines on popular classification benchmarks. Finally, to understand the potential benefit of modeling task covariates we further evaluate our method on a synthetic regression dataset.

References in corpus (10)

• Auto-Encoding Variational Bayes
• Prototypical Networks for Few-shot Learning
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• On First-Order Meta-Learning Algorithms
• Meta-Learning with Implicit Gradients
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulation
• Learning-to-Learn Stochastic Gradient Descent with Biased Regularization
• Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
• EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations