Optimizing Millions of Hyperparameters by Implicit Differentiation

Abstract

We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the proposed approach to train modern network architectures with millions of weights and millions of hyper-parameters. For example, we learn a data-augmentation network - where every weight is a hyperparameter tuned for validation performance - outputting augmented training examples. Jointly tuning weights and hyperparameters with our approach is only a few times more costly in memory and compute than standard training.

Submitted to AISTATS 2020

References in corpus (21)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Practical Bayesian Optimization of Machine Learning Algorithms
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• AutoAugment: Learning Augmentation Policies from Data
• On First-Order Meta-Learning Algorithms
• Regularizing and Optimizing LSTM Language Models
• Variational Dropout and the Local Reparameterization Trick
• Meta-Learning with Implicit Gradients
• Regularization for Deep Learning: A Taxonomy
• Generalized Inner Loop Meta-Learning
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Stochastic Hyperparameter Optimization through Hypernetworks
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• Understanding Short-Horizon Bias in Stochastic Meta-Optimization
• Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
• ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks
• Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
• Stable Opponent Shaping in Differentiable Games
• Adversarial Learning of General Transformations for Data Augmentation
• AutoAssist: A Framework to Accelerate Training of Deep Neural Networks

Cited by in corpus (13)

• Implicit differentiation of Lasso-type models for hyperparameter optimization
• HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks
• On the Iteration Complexity of Hypergradient Computation
• Meta-Learning Symmetries by Reparameterization
• Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• SecDD: Efficient and Secure Method for Remotely Training Neural Networks
• UFO-BLO: Unbiased First-Order Bilevel Optimization
• Convergence Properties of Stochastic Hypergradients
• Efficient hyperparameter optimization by way of PAC-Bayes bound minimization
• MDRefine: a Python package for refining Molecular Dynamics trajectories with experimental data
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Cost-Efficient Online Hyperparameter Optimization