On Symmetry and Initialization for Neural Networks

Abstract

This work provides an additional step in the theoretical understanding of neural networks. We consider neural networks with one hidden layer and show that when learning symmetric functions, one can choose initial conditions so that standard SGD training efficiently produces generalization guarantees. We empirically verify this and show that this does not hold when the initial conditions are chosen at random. The proof of convergence investigates the interaction between the two layers of the network. Our results highlight the importance of using symmetry in the design of neural networks.

References in corpus (12)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Large-Margin Softmax Loss for Convolutional Neural Networks
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Robust Large Margin Deep Neural Networks
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Representation Benefits of Deep Feedforward Networks
• Understanding Deep Neural Networks with Rectified Linear Units
• SGD Learns the Conjugate Kernel Class of the Network
• On the Complexity of Learning Neural Networks