Practical Gauss-Newton Optimisation for Deep Learning

Abstract

We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for which hyperparameter tuning of the optimisation parameters is often a labo- rious process, our approach can provide good performance even when used with default set- tings. A side result of our work is that for piecewise linear transfer functions, the net- work objective function can have no differ- entiable local maxima, which may partially explain why such transfer functions facilitate effective optimisation.

ICML 2017

Cited by in corpus (34)

• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Up or Down? Adaptive Rounding for Post-Training Quantization
• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
• Practical Quasi-Newton Methods for Training Deep Neural Networks
• Explicit Regularisation in Gaussian Noise Injections
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• A Theoretical Framework for Target Propagation
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• Laplace Redux -- Effortless Bayesian Deep Learning
• A literature survey of matrix methods for data science
• Deep Frank-Wolfe For Neural Network Optimization
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• Scalable Bayesian Meta-Learning through Generalized Implicit Gradients
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning
• Improving predictions of Bayesian neural nets via local linearization
• Biological credit assignment through dynamic inversion of feedforward networks
• Information Geometry of Orthogonal Initializations and Training
• Sketchy Empirical Natural Gradient Methods for Deep Learning
• DDPNOpt: Differential Dynamic Programming Neural Optimizer
• Does the Data Induce Capacity Control in Deep Learning?
• Block Mean Approximation for Efficient Second Order Optimization
• Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• Machine Learning and System Identification for Estimation in Physical Systems
• Rethinking Gauss-Newton for learning over-parameterized models
• Modular Block-diagonal Curvature Approximations for Feedforward Architectures
• Parabolic Approximation Line Search for DNNs
• TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion
• Enhance Curvature Information by Structured Stochastic Quasi-Newton Methods
• Riemannian Laplace approximations for Bayesian neural networks
• Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
• Robust Trust Region for Weakly Supervised Segmentation
• Tensor Normal Training for Deep Learning Models