Natural Neural Networks

Abstract

We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the neural network weights by implicitly whitening the representation obtained at each layer, while preserving the feed-forward computation of the network. Such networks can be trained efficiently via the proposed Projected Natural Gradient Descent algorithm (PRONG), which amortizes the cost of these reparametrizations over many parameter updates and is closely related to the Mirror Descent online learning algorithm. We highlight the benefits of our method on both unsupervised and supervised learning tasks, and showcase its scalability by training on the large-scale ImageNet Challenge dataset.

Cited by in corpus (45)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Gaussian Error Linear Units (GELUs)
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Concept Whitening for Interpretable Image Recognition
• Out-of-Distribution Generalization via Risk Extrapolation (REx)
• Sharp Minima Can Generalize For Deep Nets
• Incremental Object Detection via Meta-Learning
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Learning values across many orders of magnitude
• Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks
• Implicit Regularization in Deep Learning
• Meta-Learning with Warped Gradient Descent
• Practical Quasi-Newton Methods for Training Deep Neural Networks
• Optimization on Submanifolds of Convolution Kernels in CNNs
• Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting
• A Kronecker-factored approximate Fisher matrix for convolution layers
• Deep Frank-Wolfe For Neural Network Optimization
• Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches
• Understanding symmetries in deep networks
• Backpropagation-Friendly Eigendecomposition
• Network Deconvolution
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Symmetry-invariant optimization in deep networks
• Biological credit assignment through dynamic inversion of feedforward networks
• Regularizing activations in neural networks via distribution matching with the Wasserstein metric
• Trusting SVM for Piecewise Linear CNNs
• A Coordinate-Free Construction of Scalable Natural Gradient
• True Asymptotic Natural Gradient Optimization
• EigenGame: PCA as a Nash Equilibrium
• Data-Dependent Path Normalization in Neural Networks
• A Fully Stochastic Second-Order Trust Region Method
• Learnable Adaptive Cosine Estimator (LACE) for Image Classification
• A Neural Network model with Bidirectional Whitening
• Second-Order Neural ODE Optimizer
• First-Order Preconditioning via Hypergradient Descent
• Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs
• Meta-Learning with Hessian-Free Approach in Deep Neural Nets Training
• On Batch Orthogonalization Layers
• GeoSeq2Seq: Information Geometric Sequence-to-Sequence Networks
• Reducing the Model Order of Deep Neural Networks Using Information Theory
• Error Autocorrelation Objective Function for Improved System Modeling
• TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion
• Whitening and second order optimization both make information in the dataset unusable during training, and can reduce or prevent generalization
• Neural Random Projection: From the Initial Task To the Input Similarity Problem