Network Decoupling: From Regular to Depthwise Separable Convolutions

Abstract

Depthwise separable convolution has shown great efficiency in network design, but requires time-consuming training procedure with full training-set available. This paper first analyzes the mathematical relationship between regular convolutions and depthwise separable convolutions, and proves that the former one could be approximated with the latter one in closed form. We show depthwise separable convolutions are principal components of regular convolutions. And then we propose network decoupling (ND), a training-free method to accelerate convolutional neural networks (CNNs) by transferring pre-trained CNN models into the MobileNet-like depthwise separable convolution structure, with a promising speedup yet negligible accuracy loss. We further verify through experiments that the proposed method is orthogonal to other training-free methods like channel decomposition, spatial decomposition, etc. Combining the proposed method with them will bring even larger CNN speedup. For instance, ND itself achieves about 2X speedup for the widely used VGG16, and combined with other methods, it reaches 3.7X speedup with graceful accuracy degradation. We demonstrate that ND is widely applicable to classification networks like ResNet, and object detection network like SSD300.

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Neural Architecture Search with Reinforcement Learning
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Neural Networks with the Hashing Trick
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• Xception: Deep Learning with Depthwise Separable Convolutions
• Progressive Neural Architecture Search

Cited by in corpus (14)

• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Review: Deep Learning in Electron Microscopy
• Cluster Pruning: An Efficient Filter Pruning Method for Edge AI Vision Applications
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• Few Sample Knowledge Distillation for Efficient Network Compression
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks
• Joint Architecture and Knowledge Distillation in CNN for Chinese Text Recognition
• EEG-based Drowsiness Estimation for Driving Safety using Deep Q-Learning
• Composite Binary Decomposition Networks
• Joint Matrix Decomposition for Deep Convolutional Neural Networks Compression
• Accelerating Training using Tensor Decomposition
• Semi-tensor Product-based TensorDecomposition for Neural Network Compression
• DAC: Data-free Automatic Acceleration of Convolutional Networks