Neural Network Inference on Mobile SoCs

Abstract

The ever-increasing demand from mobile Machine Learning (ML) applications calls for evermore powerful on-chip computing resources. Mobile devices are empowered with heterogeneous multi-processor Systems-on-Chips (SoCs) to process ML workloads such as Convolutional Neural Network (CNN) inference. Mobile SoCs house several different types of ML capable components on-die, such as CPU, GPU, and accelerators. These different components are capable of independently performing inference but with very different power-performance characteristics. In this article, we provide a quantitative evaluation of the inference capabilities of the different components on mobile SoCs. We also present insights behind their respective power-performance behavior. Finally, we explore the performance limit of the mobile SoCs by synergistically engaging all the components concurrently. We observe that a mobile SoC provides up to 2x improvement with parallel inference when all its components are engaged, as opposed to engaging only one component.

Accepted to IEEE Design & Test

References in corpus (1)

• High-Throughput CNN Inference on Embedded ARM big.LITTLE Multi-Core Processors

Cited by in corpus (7)

• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• Power-Performance Characterization of TinyML Systems
• NAWQ-SR: A Hybrid-Precision NPU Engine for Efficient On-Device Super-Resolution
• PointSplit: Towards On-device 3D Object Detection with Heterogeneous Low-power Accelerators
• Optimizing DNN Inference on Multi-Accelerator SoCs at Training-time
• MEDEA: A Design-Time Multi-Objective Manager for Energy-Efficient DNN Inference on Heterogeneous Ultra-Low Power Platforms
• Comparative Study of Parameter Selection for Enhanced Edge Inference for a Multi-Output Regression model for Head Pose Estimation