Riemannian approach to batch normalization
arXiv:1709.09603
Abstract
Batch Normalization (BN) has proven to be an effective algorithm for deep neural network training by normalizing the input to each neuron and reducing the internal covariate shift. The space of weight vectors in the BN layer can be naturally interpreted as a Riemannian manifold, which is invariant to linear scaling of weights. Following the intrinsic geometry of this manifold provides a new learning rule that is more efficient and easier to analyze. We also propose intuitive and effective gradient clipping and regularization methods for the proposed algorithm by utilizing the geometry of the manifold. The resulting algorithm consistently outperforms the original BN on various types of network architectures and datasets.
to appear at NIPS 2017
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Deep Networks with Stochastic Depth
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Path-SGD: Path-Normalized Optimization in Deep Neural Networks
- Understanding symmetries in deep networks