Optimising the Performance of Convolutional Neural Networks across Computing Systems using Transfer Learning

Abstract

The choice of convolutional routines (primitives) to implement neural networks has a tremendous impact on their inference performance (execution speed) on a given hardware platform. To optimise a neural network by primitive selection, the optimal primitive is identified for each layer of the network. This process requires a lengthy profiling stage, iterating over all the available primitives for each layer configuration, to measure their execution time on the target platform. Because each primitive exploits the hardware in different ways, new profiling is needed to obtain the best performance when moving to another platform. In this work, we propose to replace this prohibitively expensive profiling stage with a machine learning based approach of performance modeling. Our approach speeds up the optimisation time drastically. After training, our performance model can estimate the performance of convolutional primitives in any layer configuration. The time to optimise the execution of large neural networks via primitive selection is reduced from hours to just seconds. Our performance model is easily transferable to other target platforms. We demonstrate this by training a performance model on an Intel platform and performing transfer learning to AMD and ARM processor devices with minimal profiled samples.

arXiv comment12 pages, 10 figures

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processors
• NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks
• Low-memory GEMM-based convolution algorithms for deep neural networks
• Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs
• Learning to infer: RL-based search for DNN primitive selection on Heterogeneous Embedded Systems
• TASO: Time and Space Optimization for Memory-Constrained DNN Inference