Meta-Learning to Improve Pre-Training

Abstract

Pre-training (PT) followed by fine-tuning (FT) is an effective method for training neural networks, and has led to significant performance improvements in many domains. PT can incorporate various design choices such as task and data reweighting strategies, augmentation policies, and noise models, all of which can significantly impact the quality of representations learned. The hyperparameters introduced by these strategies therefore must be tuned appropriately. However, setting the values of these hyperparameters is challenging. Most existing methods either struggle to scale to high dimensions, are too slow and memory-intensive, or cannot be directly applied to the two-stage PT and FT learning process. In this work, we propose an efficient, gradient-based algorithm to meta-learn PT hyperparameters. We formalize the PT hyperparameter optimization problem and propose a novel method to obtain PT hyperparameter gradients by combining implicit differentiation and backpropagation through unrolled optimization. We demonstrate that our method improves predictive performance on two real-world domains. First, we optimize high-dimensional task weighting hyperparameters for multitask pre-training on protein-protein interaction graphs and improve AUROC by up to 3.9%. Second, we optimize a data augmentation neural network for self-supervised PT with SimCLR on electrocardiography data and improve AUROC by up to 1.9%.

arXiv commentNeurIPS 2021

References in corpus (30)

• A Simple Framework for Contrastive Learning of Visual Representations
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Learning deep representations by mutual information estimation and maximization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks
• Learning Representations by Maximizing Mutual Information Across Views
• Contrastive Multiview Coding
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Similarity of Neural Network Representations Revisited
• Algorithms for Learning Kernels Based on Centered Alignment
• Meta-Learning with Implicit Gradients
• Meta-Learning with Latent Embedding Optimization
• Strategies for Pre-training Graph Neural Networks
• Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Gradient Surgery for Multi-Task Learning
• How to train your MAML
• Stochastic Hyperparameter Optimization through Hypernetworks
• Learning Data Manipulation for Augmentation and Weighting
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
• Meta-Learning Representations for Continual Learning
• Supervised Transfer Learning at Scale for Medical Imaging
• A Comprehensive Evaluation of Multi-task Learning and Multi-task Pre-training on EHR Time-series Data
• Efficient Optimization of Loops and Limits with Randomized Telescoping Sums
• Meta Approach to Data Augmentation Optimization
• Teaching with Commentaries