On-Device Neural Net Inference with Mobile GPUs

Abstract

On-device inference of machine learning models for mobile phones is desirable due to its lower latency and increased privacy. Running such a compute-intensive task solely on the mobile CPU, however, can be difficult due to limited computing power, thermal constraints, and energy consumption. App developers and researchers have begun exploiting hardware accelerators to overcome these challenges. Recently, device manufacturers are adding neural processing units into high-end phones for on-device inference, but these account for only a small fraction of hand-held devices. In this paper, we present how we leverage the mobile GPU, a ubiquitous hardware accelerator on virtually every phone, to run inference of deep neural networks in real-time for both Android and iOS devices. By describing our architecture, we also discuss how to design networks that are mobile GPU-friendly. Our state-of-the-art mobile GPU inference engine is integrated into the open-source project TensorFlow Lite and publicly available at https://tensorflow.org/lite.

arXiv commentComputer Vision and Pattern Recognition Workshop: Efficient Deep Learning for Computer Vision 2019

References in corpus (1)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Cited by in corpus (17)

• Attention Mesh: High-fidelity Face Mesh Prediction in Real-time
• Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices
• PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management
• Mind Your Weight(s): A Large-scale Study on Insufficient Machine Learning Model Protection in Mobile Apps
• Real-time Pupil Tracking from Monocular Video for Digital Puppetry
• Fast and Accurate Single-Image Depth Estimation on Mobile Devices, Mobile AI 2021 Challenge: Report
• Real-Time Video Super-Resolution on Smartphones with Deep Learning, Mobile AI 2021 Challenge: Report
• TurboTransformers: An Efficient GPU Serving System For Transformer Models
• Efficient Memory Management for Deep Neural Net Inference
• Smart at what cost? Characterising Mobile Deep Neural Networks in the wild
• ML-EXray: Visibility into ML Deployment on the Edge
• Device-Cloud Collaborative Learning for Recommendation
• BasisNet: Two-stage Model Synthesis for Efficient Inference
• Layer Pruning on Demand with Intermediate CTC
• LETI: Latency Estimation Tool and Investigation of Neural Networks inference on Mobile GPU
• An efficient and flexible inference system for serving heterogeneous ensembles of deep neural networks
• Graph-Based Fuzz Testing for Deep Learning Inference Engine