High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processors

Abstract

IoT Edge intelligence requires Convolutional Neural Network (CNN) inference to take place in the edge devices itself. ARM big.LITTLE architecture is at the heart of prevalent commercial edge devices. It comprises of single-ISA heterogeneous cores grouped into multiple homogeneous clusters that enable power and performance trade-offs. All cores are expected to be simultaneously employed in inference to attain maximal throughput. However, high communication overhead involved in parallelization of computations from convolution kernels across clusters is detrimental to throughput. We present an alternative framework called Pipe-it that employs pipelined design to split convolutional layers across clusters while limiting parallelization of their respective kernels to the assigned cluster. We develop a performance-prediction model that utilizes only the convolutional layer descriptors to predict the execution time of each layer individually on all permitted core configurations (type and count). Pipe-it then exploits the predictions to create a balanced pipeline using an efficient design space exploration algorithm. Pipe-it on average results in a 39% higher throughput than the highest antecedent throughput.

Accepted to IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems

References in corpus (10)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
• CNNdroid: GPU-Accelerated Execution of Trained Deep Convolutional Neural Networks on Android
• Optimizing CNN Model Inference on CPUs
• Modeling the Resource Requirements of Convolutional Neural Networks on Mobile Devices
• ZNN - A Fast and Scalable Algorithm for Training 3D Convolutional Networks on Multi-Core and Many-Core Shared Memory Machines
• Mobile Machine Learning Hardware at ARM: A Systems-on-Chip (SoC) Perspective
• Scheduling Computation Graphs of Deep Learning Models on Manycore CPUs
• Enabling Embedded Inference Engine with ARM Compute Library: A Case Study

Cited by in corpus (13)

• Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models
• Neural Network Inference on Mobile SoCs
• HAPI: Hardware-Aware Progressive Inference
• Power-Performance Characterization of TinyML Systems
• CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads
• Estimating the Circuit Deobfuscating Runtime based on Graph Deep Learning
• Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems
• AutoScale: Optimizing Energy Efficiency of End-to-End Edge Inference under Stochastic Variance
• Enabling Highly Efficient Capsule Networks Processing Through A PIM-Based Architecture Design
• A Framework to Enable Algorithmic Design Choice Exploration in DNNs
• Optimising the Performance of Convolutional Neural Networks across Computing Systems using Transfer Learning
• Collaborative Heterogeneous Computing on MPSoCs
• Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs