Greedy Layerwise Learning Can Scale to ImageNet

Abstract

Shallow supervised 1-hidden layer neural networks have a number of favorable properties that make them easier to interpret, analyze, and optimize than their deep counterparts, but lack their representational power. Here we use 1-hidden layer learning problems to sequentially build deep networks layer by layer, which can inherit properties from shallow networks. Contrary to previous approaches using shallow networks, we focus on problems where deep learning is reported as critical for success. We thus study CNNs on image classification tasks using the large-scale ImageNet dataset and the CIFAR-10 dataset. Using a simple set of ideas for architecture and training we find that solving sequential 1-hidden-layer auxiliary problems lead to a CNN that exceeds AlexNet performance on ImageNet. Extending this training methodology to construct individual layers by solving 2-and-3-hidden layer auxiliary problems, we obtain an 11-layer network that exceeds several members of the VGG model family on ImageNet, and can train a VGG-11 model to the same accuracy as end-to-end learning. To our knowledge, this is the first competitive alternative to end-to-end training of CNNs that can scale to ImageNet. We illustrate several interesting properties of these models theoretically and conduct a range of experiments to study the properties this training induces on the intermediate layers.

References in corpus (21)

• How transferable are features in deep neural networks?
• Wide Residual Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Understanding Neural Networks Through Deep Visualization
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Provable Bounds for Learning Some Deep Representations
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• Learning One-hidden-layer Neural Networks with Landscape Design
• AdaNet: Adaptive Structural Learning of Artificial Neural Networks
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Scattering Networks for Hybrid Representation Learning
• Decoupled Neural Interfaces using Synthetic Gradients
• On the ability of neural nets to express distributions
• Learning Deep ResNet Blocks Sequentially using Boosting Theory
• A Provably Correct Algorithm for Deep Learning that Actually Works
• Layer-wise training of deep networks using kernel similarity
• Improved Learning of One-hidden-layer Convolutional Neural Networks with Overlaps

Cited by in corpus (12)

• Shallow-Deep Networks: Understanding and Mitigating Network Overthinking
• AdaGCN: Adaboosting Graph Convolutional Networks into Deep Models
• LoCo: Local Contrastive Representation Learning
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment
• Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time
• The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods
• Learning Boolean Circuits with Neural Networks
• Contrastive Similarity Matching for Supervised Learning
• Biological Blueprints for Next Generation AI Systems
• Neighbourhood Distillation: On the benefits of non end-to-end distillation
• Associated Learning: Decomposing End-to-end Backpropagation based on Auto-encoders and Target Propagation
• Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization