Why Layer-Wise Learning is Hard to Scale-up and a Possible Solution via Accelerated Downsampling

Abstract

Layer-wise learning, as an alternative to global back-propagation, is easy to interpret, analyze, and it is memory efficient. Recent studies demonstrate that layer-wise learning can achieve state-of-the-art performance in image classification on various datasets. However, previous studies of layer-wise learning are limited to networks with simple hierarchical structures, and the performance decreases severely for deeper networks like ResNet. This paper, for the first time, reveals the fundamental reason that impedes the scale-up of layer-wise learning is due to the relatively poor separability of the feature space in shallow layers. This argument is empirically verified by controlling the intensity of the convolution operation in local layers. We discover that the poorly-separable features from shallow layers are mismatched with the strong supervision constraint throughout the entire network, making the layer-wise learning sensitive to network depth. The paper further proposes a downsampling acceleration approach to weaken the poor learning of shallow layers so as to transfer the learning emphasis to deep feature space where the separability matches better with the supervision restraint. Extensive experiments have been conducted to verify the new finding and demonstrate the advantages of the proposed downsampling acceleration in improving the performance of layer-wise learning.

References in corpus (8)

• Wide Residual Networks
• Training Neural Networks with Local Error Signals
• Highway and Residual Networks learn Unrolled Iterative Estimation
• Meta-Learning Update Rules for Unsupervised Representation Learning
• FreezeOut: Accelerate Training by Progressively Freezing Layers
• A Provably Correct Algorithm for Deep Learning that Actually Works
• Layer-wise training of deep networks using kernel similarity
• Plug-and-Play Rescaling Based Crowd Counting in Static Images