Searching for Winograd-aware Quantized Networks

Abstract

Lightweight architectural designs of Convolutional Neural Networks (CNNs) together with quantization have paved the way for the deployment of demanding computer vision applications on mobile devices. Parallel to this, alternative formulations to the convolution operation such as FFT, Strassen and Winograd, have been adapted for use in CNNs offering further speedups. Winograd convolutions are the fastest known algorithm for spatially small convolutions, but exploiting their full potential comes with the burden of numerical error, rendering them unusable in quantized contexts. In this work we propose a Winograd-aware formulation of convolution layers which exposes the numerical inaccuracies introduced by the Winograd transformations to the learning of the model parameters, enabling the design of competitive quantized models without impacting model size. We also address the source of the numerical error and propose a relaxation on the form of the transformation matrices, resulting in up to 10% higher classification accuracy on CIFAR-10. Finally, we propose wiNAS, a neural architecture search (NAS) framework that jointly optimizes a given macro-architecture for accuracy and latency leveraging Winograd-aware layers. A Winograd-aware ResNet-18 optimized with wiNAS for CIFAR-10 results in 2.66x speedup compared to im2row, one of the most widely used optimized convolution implementations, with no loss in accuracy.

arXiv commentPublished as a conference paper at MLSys 2020

References in corpus (24)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Neural Architecture Search with Reinforcement Learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DARTS: Differentiable Architecture Search
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• cuDNN: Efficient Primitives for Deep Learning
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Ternary Weight Networks
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Training Deep Nets with Sublinear Memory Cost
• Fast Training of Convolutional Networks through FFTs
• Image Super-Resolution Using Very Deep Residual Channel Attention Networks
• Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
• Visual Wake Words Dataset
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• A Study of BFLOAT16 for Deep Learning Training
• FixyNN: Efficient Hardware for Mobile Computer Vision via Transfer Learning
• Efficient Winograd Convolution via Integer Arithmetic
• StrassenNets: Deep Learning with a Multiplication Budget
• Neural networks on microcontrollers: saving memory at inference via operator reordering
• Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints

Cited by in corpus (2)

• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Fast Convolution based on Winograd Minimum Filtering: Introduction and Development