Understanding symmetries in deep networks
arXiv:1511.01029
Abstract
Recent works have highlighted scale invariance or symmetry present in the weight space of a typical deep network and the adverse effect it has on the Euclidean gradient based stochastic gradient descent optimization. In this work, we show that a commonly used deep network, which uses convolution, batch normalization, reLU, max-pooling, and sub-sampling pipeline, possess more complex forms of symmetry arising from scaling-based reparameterization of the network weights. We propose to tackle the issue of the weight space symmetry by constraining the filters to lie on the unit-norm manifold. Consequently, training the network boils down to using stochastic gradient descent updates on the unit-norm manifold. Our empirical evidence based on the MNIST dataset shows that the proposed updates improve the test performance beyond what is achieved with batch normalization and without sacrificing the computational efficiency of the weight updates.
Accepted at the 8th NIPS Workshop on Optimization for Machine Learning (OPT2015) to be held at Montreal, Canada on December 11, 2015
References in corpus (3)
Cited by in corpus (12)
- Density estimation using Real NVP
- Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
- Sharp Minima Can Generalize For Deep Nets
- Meta-Learning Mean Functions for Gaussian Processes
- Riemannian approach to batch normalization
- Symmetry-invariant optimization in deep networks
- On Stein Variational Neural Network Ensembles
- The Representation Theory of Neural Networks
- Improving Optimization for Models With Continuous Symmetry Breaking
- Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: Extension
- Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
- Are Saddles Good Enough for Deep Learning?