SparseTrain:Leveraging Dynamic Sparsity in Training DNNs on General-Purpose SIMD Processors

Abstract

Our community has greatly improved the efficiency of deep learning applications, including by exploiting sparsity in inputs. Most of that work, though, is for inference, where weight sparsity is known statically, and/or for specialized hardware. We propose a scheme to leverage dynamic sparsity during training. In particular, we exploit zeros introduced by the ReLU activation function to both feature maps and their gradients. This is challenging because the sparsity degree is moderate and the locations of zeros change over time. We also rely purely on software. We identify zeros in a dense data representation without transforming the data and performs conventional vectorized computation. Variations of the scheme are applicable to all major components of training: forward propagation, backward propagation by inputs, and backward propagation by weights. Our method significantly outperforms a highly-optimized dense direct convolution on several popular deep neural networks. At realistic sparsity, we speed up the training of the non-initial convolutional layers in VGG16, ResNet-34, ResNet-50, and Fixup ResNet-50 by 2.19x, 1.37x, 1.31x, and 1.51x respectively on an Intel Skylake-X CPU.

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Trained Ternary Quantization
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration