Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions

Abstract

Neural networks rely on convolutions to aggregate spatial information. However, spatial convolutions are expensive in terms of model size and computation, both of which grow quadratically with respect to kernel size. In this paper, we present a parameter-free, FLOP-free "shift" operation as an alternative to spatial convolutions. We fuse shifts and point-wise convolutions to construct end-to-end trainable shift-based modules, with a hyperparameter characterizing the tradeoff between accuracy and efficiency. To demonstrate the operation's efficacy, we replace ResNet's 3x3 convolutions with shift-based modules for improved CIFAR10 and CIFAR100 accuracy using 60% fewer parameters; we additionally demonstrate the operation's resilience to parameter reduction on ImageNet, outperforming ResNet family members. We finally show the shift operation's applicability across domains, achieving strong performance with fewer parameters on classification, face verification and style transfer.

Source code will be released afterwards

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Aggregated Residual Transformations for Deep Neural Networks
• Xception: Deep Learning with Depthwise Separable Convolutions

Cited by in corpus (17)

• TSM: Temporal Shift Module for Efficient Video Understanding
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Dynamic Convolution: Attention over Convolution Kernels
• Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization
• SqueezeWave: Extremely Lightweight Vocoders for On-device Speech Synthesis
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• Shift-based Primitives for Efficient Convolutional Neural Networks
• Efficient Deep Neural Networks
• Learning Student Networks via Feature Embedding
• DARC: Differentiable ARchitecture Compression
• Motion Feature Network: Fixed Motion Filter for Action Recognition
• Pruning Filter in Filter
• Diagonalwise Refactorization: An Efficient Training Method for Depthwise Convolutions
• Ultra Power-Efficient CNN Domain Specific Accelerator with 9.3TOPS/Watt for Mobile and Embedded Applications
• Accelerating Training using Tensor Decomposition
• Full-stack Optimization for Accelerating CNNs with FPGA Validation