A Provably Correct Algorithm for Deep Learning that Actually Works

Abstract

We describe a layer-by-layer algorithm for training deep convolutional networks, where each step involves gradient updates for a two layer network followed by a simple clustering algorithm. Our algorithm stems from a deep generative model that generates mages level by level, where lower resolution images correspond to latent semantic classes. We analyze the convergence rate of our algorithm assuming that the data is indeed generated according to this model (as well as additional assumptions). While we do not pretend to claim that the assumptions are realistic for natural images, we do believe that they capture some true properties of real data. Furthermore, we show that our algorithm actually works in practice (on the CIFAR dataset), achieving results in the same ballpark as that of vanilla convolutional neural networks that are being trained by stochastic gradient descent. Finally, our proof techniques may be of independent interest.

References in corpus (3)

• Recovery Guarantees for One-hidden-layer Neural Networks
• When is a Convolutional Filter Easy To Learn?
• SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Data

Cited by in corpus (9)

• Revisiting Locally Supervised Learning: an Alternative to End-to-end Training
• Greedy Layerwise Learning Can Scale to ImageNet
• Computational Separation Between Convolutional and Fully-Connected Networks
• End-to-end Learning of a Convolutional Neural Network via Deep Tensor Decomposition
• DANTE: Deep AlterNations for Training nEural networks
• Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsampling
• Learning Boolean Circuits with Neural Networks
• The staircase property: How hierarchical structure can guide deep learning
• From Boltzmann Machines to Neural Networks and Back Again