Structured Convolutions for Efficient Neural Network Design

Abstract

In this work, we tackle model efficiency by exploiting redundancy in the \textit{implicit structure} of the building blocks of convolutional neural networks. We start our analysis by introducing a general definition of Composite Kernel structures that enable the execution of convolution operations in the form of efficient, scaled, sum-pooling components. As its special case, we propose \textit{Structured Convolutions} and show that these allow decomposition of the convolution operation into a sum-pooling operation followed by a convolution with significantly lower complexity and fewer weights. We show how this decomposition can be applied to 2D and 3D kernels as well as the fully-connected layers. Furthermore, we present a Structural Regularization loss that promotes neural network layers to leverage on this desired structure in a way that, after training, they can be decomposed with negligible performance loss. By applying our method to a wide range of CNN architectures, we demonstrate "structured" versions of the ResNets that are up to 2$\times$ smaller and a new Structured-MobileNetV2 that is more efficient while staying within an accuracy loss of 1% on ImageNet and CIFAR-10 datasets. We also show similar structured versions of EfficientNet on ImageNet and HRNet architecture for semantic segmentation on the Cityscapes dataset. Our method performs equally well or superior in terms of the complexity reduction in comparison to the existing tensor decomposition and channel pruning methods.

arXiv commentCamera-ready for NeurIPS 2020

References in corpus (22)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Rethinking the Value of Network Pruning
• Ternary Weight Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Convolutional neural networks with low-rank regularization
• Deep High-Resolution Representation Learning for Visual Recognition
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Slimmable Neural Networks
• GhostNet: More Features from Cheap Operations
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Tensor-Train Recurrent Neural Networks for Video Classification
• Automated Pruning for Deep Neural Network Compression
• DCFNet: Deep Neural Network with Decomposed Convolutional Filters
• Learned Threshold Pruning
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• AdderNet: Do We Really Need Multiplications in Deep Learning?
• Shift-based Primitives for Efficient Convolutional Neural Networks
• BasisConv: A method for compressed representation and learning in CNNs