Meta-Learning Update Rules for Unsupervised Representation Learning

Abstract

A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this involves minimizing a surrogate objective, such as the negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as a side effect. In this work, we propose instead to directly target later desired tasks by meta-learning an unsupervised learning rule which leads to representations useful for those tasks. Specifically, we target semi-supervised classification performance, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations useful for this task. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function, which enables it to generalize to different neural network architectures, datasets, and data modalities. We show that the meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We further show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths, and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from image datasets to a text task.

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Practical Bayesian Optimization of Machine Learning Algorithms
• Prototypical Networks for Few-shot Learning
• Playing Atari with Deep Reinforcement Learning
• Neural Architecture Search with Reinforcement Learning
• On the difficulty of training Recurrent Neural Networks
• A Simple Neural Attentive Meta-Learner
• Searching for Activation Functions
• Meta-Learning for Semi-Supervised Few-Shot Classification
• Adversarial Feature Learning
• Adversarially Learned Inference
• Designing Neural Network Architectures using Reinforcement Learning
• Unsupervised Learning by Predicting Noise
• Neural Optimizer Search with Reinforcement Learning
• Building high-level features using large scale unsupervised learning
• Learning to Learn without Gradient Descent by Gradient Descent
• Unsupervised Learning via Meta-Learning
• Evolved Policy Gradients
• Learned Optimizers that Scale and Generalize
• Time-Contrastive Networks: Self-Supervised Learning from Video
• Supervising Unsupervised Learning
• Truncated Back-propagation for Bilevel Optimization

Cited by in corpus (32)

• Artificial neural networks for neuroscientists: A primer
• An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning
• Evolved Policy Gradients
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• Zero-shot Text Classification With Generative Language Models
• Meta-Learning with Warped Gradient Descent
• Unsupervised Meta-Learning for Reinforcement Learning
• Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data
• Meta-Learning Requires Meta-Augmentation
• DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Meta-Learning Representations for Continual Learning
• LoCo: Local Contrastive Representation Learning
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Revisiting Meta-Learning as Supervised Learning
• How to Train Your MAML to Excel in Few-Shot Classification
• When MAML Can Adapt Fast and How to Assist When It Cannot
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption
• Meta-Amortized Variational Inference and Learning
• Parallel Training of Deep Networks with Local Updates
• Learning to Learn with Feedback and Local Plasticity
• Formulating Camera-Adaptive Color Constancy as a Few-shot Meta-Learning Problem
• Learning to Transfer: Unsupervised Meta Domain Translation
• What Can Learned Intrinsic Rewards Capture?
• Credit Assignment Through Broadcasting a Global Error Vector
• A contrastive rule for meta-learning
• MPLP: Learning a Message Passing Learning Protocol
• Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsampling
• Theoretical bounds on estimation error for meta-learning
• Effective Regularization Through Loss-Function Metalearning
• Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative