Diagonal Rescaling For Neural Networks

Abstract

We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations. Investigating why this algorithm lacks in robustness then reveals two interesting insights. The first insight suggests a new way to scale the stepsizes, clarifying popular algorithms such as RMSProp as well as old neural network tricks such as fanin stepsize scaling. The second insight stresses the practical importance of dealing with fast changes of the curvature of the cost.

References in corpus (3)

• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• Practical Riemannian Neural Networks

Cited by in corpus (5)

• Large Batch Training of Convolutional Networks
• Rotate your Networks: Better Weight Consolidation and Less Catastrophic Forgetting
• Linear Convergence of Adaptive Stochastic Gradient Descent
• Fast Approximation of the Gauss-Newton Hessian Matrix for the Multilayer Perceptron
• Train Feedfoward Neural Network with Layer-wise Adaptive Rate via Approximating Back-matching Propagation