How Not to Give a FLOP: Combining Regularization and Pruning for Efficient Inference

Abstract

The challenge of speeding up deep learning models during the deployment phase has been a large, expensive bottleneck in the modern tech industry. In this paper, we examine the use of both regularization and pruning for reduced computational complexity and more efficient inference in Deep Neural Networks (DNNs). In particular, we apply mixup and cutout regularizations and soft filter pruning to the ResNet architecture, focusing on minimizing floating-point operations (FLOPs). Furthermore, by using regularization in conjunction with network pruning, we show that such a combination makes a substantial improvement over each of the two techniques individually.

Citations added, typos fixed

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Improved Regularization of Convolutional Neural Networks with Cutout
• Compressing Deep Convolutional Networks using Vector Quantization
• Trained Ternary Quantization
• Pruning Filters for Efficient ConvNets
• Memory Bounded Deep Convolutional Networks