Approximate Random Dropout

Abstract

The training phases of Deep neural network~(DNN) consumes enormous processing time and energy. Compression techniques utilizing the sparsity of DNNs can effectively accelerate the inference phase of DNNs. However, it can be hardly used in the training phase because the training phase involves dense matrix-multiplication using General Purpose Computation on Graphics Processors (GPGPU), which endorse regular and structural data layout. In this paper, we propose the Approximate Random Dropout that replaces the conventional random dropout of neurons and synapses with a regular and predefined patterns to eliminate the unnecessary computation and data access. To compensate the potential performance loss we develop a SGD-based Search Algorithm to produce the distribution of dropout patterns. We prove our approach is statistically equivalent to the previous dropout method. Experiments results on MLP and LSTM using well-known benchmarks show that the proposed Approximate Random Dropout can reduce the training time by $20\%$-$77\%$ ($19\%$-$60\%$) when dropout rate is $0.3$-$0.7$ on MLP (LSTM) with marginal accuracy drop.

arXiv comment7 pages, 6 figures, conference

References in corpus (15)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Recent Trends in Deep Learning Based Natural Language Processing
• Compressing Deep Convolutional Networks using Vector Quantization
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Learning Structured Sparsity in Deep Neural Networks
• Staleness-aware Async-SGD for Distributed Deep Learning
• Texygen: A Benchmarking Platform for Text Generation Models
• Learning Intrinsic Sparse Structures within Long Short-Term Memory