Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions

Abstract

Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters to optimal weights and biases. We show how to construct scalable best-response approximations for neural networks by modeling the best-response as a single network whose hidden units are gated conditionally on the regularizer. We justify this approximation by showing the exact best-response for a shallow linear network with L2-regularized Jacobian can be represented by a similar gating mechanism. We fit this model using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function. Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters, and dropout probabilities. Because the hyperparameters are adapted online, our approach discovers hyperparameter schedules that can outperform fixed hyperparameter values. Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems. We call our networks, which update their own hyperparameters online during training, Self-Tuning Networks (STNs).

Published as a conference paper at ICLR 2019

Cited by in corpus (17)

• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Learning Deformable Image Registration from Optimization: Perspective, Modules, Bilevel Training and Beyond
• Learning the Pareto Front with Hypernetworks
• Controllable Pareto Multi-Task Learning
• Gradient Descent-Ascent Provably Converges to Strict Local Minmax Equilibria with a Finite Timescale Separation
• Gradient-based algorithms for multi-objective bi-level optimization
• Teaching with Commentaries
• A Value-Function-based Interior-point Method for Non-convex Bi-level Optimization
• Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
• Adversarially Robust Learning for Security-Constrained Optimal Power Flow
• Theoretical bounds on estimation error for meta-learning
• Hypernetwork-Based Augmentation
• Benefiting Deep Latent Variable Models via Learning the Prior and Removing Latent Regularization
• Stabilizing Bi-Level Hyperparameter Optimization using Moreau-Yosida Regularization
• Meta-Learning to Improve Pre-Training
• Complex Momentum for Optimization in Games
• Automated Graph Learning via Population Based Self-Tuning GCN