Gradient-based Hyperparameter Optimization through Reversible Learning

Abstract

Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyperparameters, including step-size and momentum schedules, weight initialization distributions, richly parameterized regularization schemes, and neural network architectures. We compute hyperparameter gradients by exactly reversing the dynamics of stochastic gradient descent with momentum.

10 figures. Submitted to ICML

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Theano: new features and speed improvements
• Multi-task Neural Networks for QSAR Predictions
• Nested Variational Compression in Deep Gaussian Processes
• Automatic Differentiation of Algorithms for Machine Learning

Cited by in corpus (86)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• AutoML: A Survey of the State-of-the-Art
• A Closer Look at Memorization in Deep Networks
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Learning Discrete Structures for Graph Neural Networks
• Learning to Optimize
• Hyperparameter optimization with approximate gradient
• Automated Machine Learning in Practice: State of the Art and Recent Results
• Bilevel Programming for Hyperparameter Optimization and Meta-Learning
• Programming with a Differentiable Forth Interpreter
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Stochastic Hyperparameter Optimization through Hypernetworks
• Learning to Optimize Neural Nets
• Online Learning Rate Adaptation with Hypergradient Descent
• From Learning to Meta-Learning: Reduced Training Overhead and Complexity for Communication Systems
• Variational Inference using Implicit Distributions
• Forward and Reverse Gradient-Based Hyperparameter Optimization
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• Learning the Effect of Registration Hyperparameters with HyperMorph
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Automatic differentiation of dominant eigensolver and its applications in quantum physics
• Semi-Amortized Variational Autoencoders
• One-Shot Learning of Stochastic Differential Equations with Data Adapted Kernels
• AutoLoss: Learning Discrete Schedules for Alternate Optimization
• Hyperparameter Optimization: A Spectral Approach
• End-to-End Learning for Structured Prediction Energy Networks
• Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back
• HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks
• BlinkML: Efficient Maximum Likelihood Estimation with Probabilistic Guarantees
• Communication-Efficient Robust Federated Learning with Noisy Labels
• Regularization Learning Networks: Deep Learning for Tabular Datasets
• Robust MAML: Prioritization task buffer with adaptive learning process for model-agnostic meta-learning
• Differentiable Neural Input Search for Recommender Systems
• TherML: Thermodynamics of Machine Learning
• Confusion of Tagged Perturbations in Forward Automatic Differentiation of Higher-Order Functions
• Learning Optimal Representations with the Decodable Information Bottleneck
• An Introduction to Deep Learning for the Physical Layer
• Bayesian Active Meta-Learning for Few Pilot Demodulation and Equalization
• Designing Perceptual Puzzles by Differentiating Probabilistic Programs
• Flexible Dataset Distillation: Learn Labels Instead of Images
• Comparative Evaluation of Metaheuristic Algorithms for Hyperparameter Selection in Short-Term Weather Forecasting
• How to decay your learning rate
• Representing smooth functions as compositions of near-identity functions with implications for deep network optimization
• Reinforcement Learning for Learning Rate Control
• Structured Prediction Energy Networks
• Early Stopping is Nonparametric Variational Inference
• Deep Q-Networks for Accelerating the Training of Deep Neural Networks
• Nonparametric Neural Networks
• Predict and Constrain: Modeling Cardinality in Deep Structured Prediction
• A Batched Scalable Multi-Objective Bayesian Optimization Algorithm
• Evolutionary Architecture Search For Deep Multitask Networks
• DHA: End-to-End Joint Optimization of Data Augmentation Policy, Hyper-parameter and Architecture
• Differentiable Rendering with Perturbed Optimizers
• Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters
• Tricks from Deep Learning
• A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning
• Meta Approach to Data Augmentation Optimization
• INR-Arch: A Dataflow Architecture and Compiler for Arbitrary-Order Gradient Computations in Implicit Neural Representation Processing
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
• MLtuner: System Support for Automatic Machine Learning Tuning
• Rectified Meta-Learning from Noisy Labels for Robust Image-based Plant Disease Diagnosis
• Efficient Online Hyperparameter Optimization for Kernel Ridge Regression with Applications to Traffic Time Series Prediction
• EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization
• DiffSharp: Automatic Differentiation Library
• Discretization-free Knowledge Gradient Methods for Bayesian Optimization
• A Generalizable Approach to Learning Optimizers
• Deep Bilevel Learning
• Dynamic Automatic Differentiation of GPU Broadcast Kernels
• Regularized deep learning with nonconvex penalties
• Differential Evolution Algorithm based Hyper-Parameters Selection of Convolutional Neural Network for Speech Command Recognition
• Deep Learning Application in Security and Privacy -- Theory and Practice: A Position Paper
• First-Order Preconditioning via Hypergradient Descent
• Neural Generative Models for Global Optimization with Gradients
• Model-Agnostic Meta-Learning using Runge-Kutta Methods
• Gradient-based Regularization Parameter Selection for Problems with Non-smooth Penalty Functions
• Optimizing for Generalization in Machine Learning with Cross-Validation Gradients
• Deep Inverse Optimization
• Optimizing Recurrent Neural Networks Architectures under Time Constraints
• Interactive Agent Modeling by Learning to Probe
• On the complexity of nonsmooth automatic differentiation
• Optimizing Hyperparameters in CNNs using Bilevel Programming in Time Series Data
• Relax and penalize: a new bilevel approach to mixed-binary hyperparameter optimization
• Gradient-based Hyperparameter Optimization Over Long Horizons
• Far-HO: A Bilevel Programming Package for Hyperparameter Optimization and Meta-Learning
• Predicting the success of Gradient Descent for a particular Dataset-Architecture-Initialization (DAI)