ISP4ML: Understanding the Role of Image Signal Processing in Efficient Deep Learning Vision Systems
arXiv:1911.07954
Abstract
Convolutional neural networks (CNNs) are now predominant components in a variety of computer vision (CV) systems. These systems typically include an image signal processor (ISP), even though the ISP is traditionally designed to produce images that look appealing to humans. In CV systems, it is not clear what the role of the ISP is, or if it is even required at all for accurate prediction. In this work, we investigate the efficacy of the ISP in CNN classification tasks, and outline the system-level trade-offs between prediction accuracy and computational cost. To do so, we build software models of a configurable ISP and an imaging sensor in order to train CNNs on ImageNet with a range of different ISP settings and functionality. Results on ImageNet show that an ISP improves accuracy by 4.6%-12.2% on MobileNet architectures of different widths. Results using ResNets demonstrate that these trends also generalize to deeper networks. An ablation study of the various processing stages in a typical ISP reveals that the tone mapper is the most significant stage when operating on high dynamic range (HDR) images, by providing 5.8% average accuracy improvement alone. Overall, the ISP benefits system efficiency because the memory and computational costs of the ISP is minimal compared to the cost of using a larger CNN to achieve the same accuracy.
arXiv comment13 pages, 11 figures
References in corpus (9)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- DARTS: Differentiable Architecture Search
- ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
- DeepISP: Towards Learning an End-to-End Image Processing Pipeline
- SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
- Learning Deep Convolutional Networks for Demosaicing
- FixyNN: Efficient Hardware for Mobile Computer Vision via Transfer Learning