Automated Design Space Exploration for optimised Deployment of DNN on Arm Cortex-A CPUs
arXiv:2006.05181 · doi:10.1109/TCAD.2020.3046568
Abstract
The spread of deep learning on embedded devices has prompted the development of numerous methods to optimise the deployment of deep neural networks (DNN). Works have mainly focused on: i) efficient DNN architectures, ii) network optimisation techniques such as pruning and quantisation, iii) optimised algorithms to speed up the execution of the most computational intensive layers and, iv) dedicated hardware to accelerate the data flow and computation. However, there is a lack of research on cross-level optimisation as the space of approaches becomes too large to test and obtain a globally optimised solution. Thus, leading to suboptimal deployment in terms of latency, accuracy, and memory. In this work, we first detail and analyse the methods to improve the deployment of DNNs across the different levels of software optimisation. Building on this knowledge, we present an automated exploration framework to ease the deployment of DNNs. The framework relies on a Reinforcement Learning search that, combined with a deep learning inference framework, automatically explores the design space and learns an optimised solution that speeds up the performance and reduces the memory on embedded CPU platforms. Thus, we present a set of results for state-of-the-art DNNs on a range of Arm Cortex-A CPU platforms achieving up to 4x improvement in performance and over 2x reduction in memory with negligible loss in accuracy with respect to the BLAS floating-point implementation.
References in corpus (23)
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- cuDNN: Efficient Primitives for Deep Learning
- Quantizing deep convolutional networks for efficient inference: A whitepaper
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Fixed Point Quantization of Deep Convolutional Networks
- Deep Reinforcement Learning: An Overview
- Designing Neural Network Architectures using Reinforcement Learning
- Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
- Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
- Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
- Moonshine: Distilling with Cheap Convolutions
- SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
- Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
- And the Bit Goes Down: Revisiting the Quantization of Neural Networks
- Low-memory GEMM-based convolution algorithms for deep neural networks
- High Performance Zero-Memory Overhead Direct Convolutions
- The Indirect Convolution Algorithm
- QUENN: QUantization Engine for low-power Neural Networks
- Learning to infer: RL-based search for DNN primitive selection on Heterogeneous Embedded Systems