Modulating Regularization Frequency for Efficient Compression-Aware Model Training

Abstract

While model compression is increasingly important because of large neural network size, compression-aware training is challenging as it needs sophisticated model modifications and longer training time.In this paper, we introduce regularization frequency (i.e., how often compression is performed during training) as a new regularization technique for a practical and efficient compression-aware training method. For various regularization techniques, such as weight decay and dropout, optimizing the regularization strength is crucial to improve generalization in Deep Neural Networks (DNNs). While model compression also demands the right amount of regularization, the regularization strength incurred by model compression has been controlled only by compression ratio. Throughout various experiments, we show that regularization frequency critically affects the regularization strength of model compression. Combining regularization frequency and compression ratio, the amount of weight updates by model compression per mini-batch can be optimized to achieve the best model accuracy. Modulating regularization frequency is implemented by occasional model compression while conventional compression-aware training is usually performed for every mini-batch.

arXiv commentarXiv admin note: text overlap with arXiv:1905.10145

References in corpus (19)

• Decoupled Weight Decay Regularization
• An overview of gradient descent optimization algorithms
• Recurrent Neural Network Regularization
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• cuDNN: Efficient Primitives for Deep Learning
• Trained Ternary Quantization
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Revisiting Small Batch Training for Deep Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• L2 Regularization versus Batch and Weight Normalization
• TensorLy: Tensor Learning in Python
• Stabilizing the Lottery Ticket Hypothesis
• Alternating Multi-bit Quantization for Recurrent Neural Networks
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• Three Mechanisms of Weight Decay Regularization
• MEC: Memory-efficient Convolution for Deep Neural Network
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Stochastic Model Pruning via Weight Dropping Away and Back