Provable Methods for Training Neural Networks with Sparse Connectivity

Abstract

We provide novel guaranteed approaches for training feedforward neural networks with sparse connectivity. We leverage on the techniques developed previously for learning linear networks and show that they can also be effectively adopted to learn non-linear networks. We operate on the moments involving label and the score function of the input, and show that their factorization provably yields the weight matrix of the first layer of a deep network under mild conditions. In practice, the output of our method can be employed as effective initializers for gradient descent.

Accepted for presentation at Neural Information Processing Systems(NIPS) 2014 Deep Learning workshop and Accepted as a workshop contribution at ICLR 2015

References in corpus (2)

• What Regularized Auto-Encoders Learn from the Data Generating Distribution
• Clustering via Mode Seeking by Direct Estimation of the Gradient of a Log-Density

Cited by in corpus (5)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Depth with Nonlinearity Creates No Bad Local Minima in ResNets
• Learning Neural Networks with Two Nonlinear Layers in Polynomial Time
• Learning Two Layer Rectified Neural Networks in Polynomial Time
• Eigenvalue Decay Implies Polynomial-Time Learnability for Neural Networks