ISyNet: Convolutional Neural Networks design for AI accelerator
arXiv:2109.01932
Abstract
In recent years Deep Learning reached significant results in many practical problems, such as computer vision, natural language processing, speech recognition and many others. For many years the main goal of the research was to improve the quality of models, even if the complexity was impractically high. However, for the production solutions, which often require real-time work, the latency of the model plays a very important role. Current state-of-the-art architectures are found with neural architecture search (NAS) taking model complexity into account. However, designing of the search space suitable for specific hardware is still a challenging task. To address this problem we propose a measure of hardware efficiency of neural architecture search space - matrix efficiency measure (MEM); a search space comprising of hardware-efficient operations; a latency-aware scaling method; and ISyNet - a set of architectures designed to be fast on the specialized neural processing unit (NPU) hardware and accurate at the same time. We show the advantage of the designed architectures for the NPU devices on ImageNet and the generalization ability for the downstream classification and detection tasks.
arXiv comment13 pages, 5 figures
References in corpus (14)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- YOLOv3: An Incremental Improvement
- Deep Residual Learning for Image Recognition
- Neural Architecture Search with Reinforcement Learning
- Gaussian Error Linear Units (GELUs)
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Searching for Activation Functions
- AutoAugment: Learning Augmentation Policies from Data
- Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
- Accelerator-aware Neural Network Design using AutoML
- Deep Mutual Learning
- Sample-Efficient Neural Architecture Search by Learning Action Space