Uniform Priors for Data-Efficient Transfer

Abstract

Deep Neural Networks have shown great promise on a variety of downstream applications; but their ability to adapt and generalize to new data and tasks remains a challenge. However, the ability to perform few or zero-shot adaptation to novel tasks is important for the scalability and deployment of machine learning models. It is therefore crucial to understand what makes for good, transfer-able features in deep networks that best allow for such adaptation. In this paper, we shed light on this by showing that features that are most transferable have high uniformity in the embedding space and propose a uniformity regularization scheme that encourages better transfer and feature reuse. We evaluate the regularization on its ability to facilitate adaptation to unseen tasks and data, for which we conduct a thorough experimental study covering four relevant, and distinct domains: few-shot Meta-Learning, Deep Metric Learning, Zero-Shot Domain Adaptation, as well as Out-of-Distribution classification. Across all experiments, we show that uniformity regularization consistently offers benefits over baseline methods and is able to achieve state-of-the-art performance in Deep Metric Learning and Meta-Learning.

References in corpus (31)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Language Models are Few-Shot Learners
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Wide Residual Networks
• Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
• Deep Neural Networks as Gaussian Processes
• Contrastive Multiview Coding
• Deep CORAL: Correlation Alignment for Deep Domain Adaptation
• Invariant Risk Minimization
• Meta-Learning with Implicit Gradients
• Wasserstein Auto-Encoders
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Unsupervised Learning via Meta-Learning
• Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML
• SoftTriple Loss: Deep Metric Learning Without Triplet Sampling
• Revisiting Training Strategies and Generalization Performance in Deep Metric Learning
• Meta-Learning without Memorization
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Unsupervised Meta-Learning For Few-Shot Image Classification
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Classification is a Strong Baseline for Deep Metric Learning
• Torchmeta: A Meta-Learning library for PyTorch
• Unsupervised Curricula for Visual Meta-Reinforcement Learning
• On Adversarial Mixup Resynthesis
• SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
• Task-Aware Variational Adversarial Active Learning
• Diversity inducing Information Bottleneck in Model Ensembles
• Unsupervised Deep Metric Learning via Auxiliary Rotation Loss