Separable Layers Enable Structured Efficient Linear Substitutions
arXiv:1906.00859
Abstract
In response to the development of recent efficient dense layers, this paper shows that something as simple as replacing linear components in pointwise convolutions with structured linear decompositions also produces substantial gains in the efficiency/accuracy tradeoff. Pointwise convolutions are fully connected layers and are thus prepared for replacement by structured transforms. Networks using such layers are able to learn the same tasks as those using standard convolutions, and provide Pareto-optimal benefits in efficiency/accuracy, both in terms of computation (mult-adds) and parameter count (and hence memory). Code is available at https://github.com/BayesWatch/deficient-efficient.
References in corpus (12)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Improved Regularization of Convolutional Neural Networks with Cutout
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- DARTS: Differentiable Architecture Search
- Compressing Neural Networks with the Hashing Trick
- Learning the Number of Neurons in Deep Networks
- Structured Transforms for Small-Footprint Deep Learning
- Moonshine: Distilling with Cheap Convolutions
- ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
- Structured adaptive and random spinners for fast machine learning computations
- Understanding and Training Deep Diagonal Circulant Neural Networks
- CircConv: A Structured Convolution with Low Complexity