PatDNN: Achieving Real-Time DNN Execution on Mobile Devices with Pattern-based Weight Pruning

Abstract

With the emergence of a spectrum of high-end mobile devices, many applications that formerly required desktop-level computation capability are being transferred to these devices. However, executing the inference of Deep Neural Networks (DNNs) is still challenging considering high computation and storage demands, specifically, if real-time performance with high accuracy is needed. Weight pruning of DNNs is proposed, but existing schemes represent two extremes in the design space: non-structured pruning is fine-grained, accurate, but not hardware friendly; structured pruning is coarse-grained, hardware-efficient, but with higher accuracy loss. In this paper, we introduce a new dimension, fine-grained pruning patterns inside the coarse-grained structures, revealing a previously unknown point in design space. With the higher accuracy enabled by fine-grained pruning patterns, the unique insight is to use the compiler to re-gain and guarantee high hardware efficiency. In other words, our method achieves the best of both worlds, and is desirable across theory/algorithm, compiler, and hardware levels. The proposed PatDNN is an end-to-end framework to efficiently execute DNN on mobile devices with the help of a novel model compression technique (pattern-based pruning based on extended ADMM solution framework) and a set of thorough architecture-aware compiler- and code generation-based optimizations (filter kernel reordering, compressed weight storage, register load redundancy elimination, and parameter auto-tuning). Evaluation results demonstrate that PatDNN outperforms three state-of-the-art end-to-end DNN frameworks, TensorFlow Lite, TVM, and Alibaba Mobile Neural Network with speedup up to 44.5x, 11.4x, and 7.1x, respectively, with no accuracy compromise. Real-time inference of representative large-scale DNNs (e.g., VGG-16, ResNet-50) can be achieved using mobile devices.

To be published in the Proceedings of Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 20)

References in corpus (1)

• DeepCache: Principled Cache for Mobile Deep Vision

Cited by in corpus (14)

• Slim-neck by GSConv: A lightweight-design for real-time detector architectures
• Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI
• Hardware-Aware DNN Compression via Diverse Pruning and Mixed-Precision Quantization
• HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity
• Efficient N:M Sparse DNN Training Using Algorithm, Architecture, and Dataflow Co-Design
• ClickTrain: Efficient and Accurate End-to-End Deep Learning Training via Fine-Grained Architecture-Preserving Pruning
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• SoD${}^2$: Statically Optimizing Dynamic Deep Neural Network
• AdapMTL: Adaptive Pruning Framework for Multitask Learning Model
• MoViDNN: A Mobile Platform for Evaluating Video Quality Enhancement with Deep Neural Networks
• Sparse Periodic Systolic Dataflow for Lowering Latency and Power Dissipation of Convolutional Neural Network Accelerators
• Mobiprox: Supporting Dynamic Approximate Computing on Mobiles