PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration

Abstract

State-of-the-art convolutional neural networks (CNNs) used in vision applications have large models with numerous weights. Training these models is very compute- and memory-resource intensive. Much research has been done on pruning or compressing these models to reduce the cost of inference, but little work has addressed the costs of training. We focus precisely on accelerating training. We propose PruneTrain, a cost-efficient mechanism that gradually reduces the training cost during training. PruneTrain uses a structured group-lasso regularization approach that drives the training optimization toward both high accuracy and small weight values. Small weights can then be periodically removed by reconfiguring the network model to a smaller one. By using a structured-pruning approach and additional reconfiguration techniques we introduce, the pruned model can still be efficiently processed on a GPU accelerator. Overall, PruneTrain achieves a reduction of 39% in the end-to-end training time of ResNet50 for ImageNet by reducing computation cost by 40% in FLOPs, memory accesses by 37% for memory bandwidth bound layers, and the inter-accelerator communication by 55%.

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Channel Pruning for Accelerating Very Deep Neural Networks
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Norm matters: efficient and accurate normalization schemes in deep networks
• Mini-batch Serialization: CNN Training with Inter-layer Data Reuse

Cited by in corpus (6)

• ElasticTrainer: Speeding Up On-Device Training with Runtime Elastic Tensor Selection
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks
• CSB-RNN: A Faster-than-Realtime RNN Acceleration Framework with Compressed Structured Blocks
• ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning
• SparseTrain:Leveraging Dynamic Sparsity in Training DNNs on General-Purpose SIMD Processors
• FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training