EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices

Abstract

In recent years, advances in deep learning have resulted in unprecedented leaps in diverse tasks spanning from speech and object recognition to context awareness and health monitoring. As a result, an increasing number of AI-enabled applications are being developed targeting ubiquitous and mobile devices. While deep neural networks (DNNs) are getting bigger and more complex, they also impose a heavy computational and energy burden on the host devices, which has led to the integration of various specialized processors in commodity devices. Given the broad range of competing DNN architectures and the heterogeneity of the target hardware, there is an emerging need to understand the compatibility between DNN-platform pairs and the expected performance benefits on each platform. This work attempts to demystify this landscape by systematically evaluating a collection of state-of-the-art DNNs on a wide variety of commodity devices. In this respect, we identify potential bottlenecks in each architecture and provide important guidelines that can assist the community in the co-design of more efficient DNNs and accelerators.

arXiv commentAccepted at MobiSys 2019: 3rd International Workshop on Embedded and Mobile Deep Learning (EMDL), 2019

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Benchmark Analysis of Representative Deep Neural Network Architectures
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
• Moonshine: Distilling with Cheap Convolutions
• pCAMP: Performance Comparison of Machine Learning Packages on the Edges

Cited by in corpus (15)

• SPINN: Synergistic Progressive Inference of Neural Networks over Device and Cloud
• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• MobiSR: Efficient On-Device Super-Resolution through Heterogeneous Mobile Processors
• Deep Learning-Based Anomaly Detection in Cyber-Physical Systems: Progress and Opportunities
• DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device
• It's always personal: Using Early Exits for Efficient On-Device CNN Personalisation
• ANNETTE: Accurate Neural Network Execution Time Estimation with Stacked Models
• Exploring Optimal Deep Learning Models for Image-based Malware Variant Classification
• NAWQ-SR: A Hybrid-Precision NPU Engine for Efficient On-Device Super-Resolution
• Smart at what cost? Characterising Mobile Deep Neural Networks in the wild
• CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads
• Mobiprox: Supporting Dynamic Approximate Computing on Mobiles
• A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities
• Neural Enhancement in Content Delivery Systems: The State-of-the-Art and Future Directions
• Performance landscape of resource-constrained platforms targeting DNNs