Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices
arXiv:2005.05085
Abstract
Due to increasing amounts of data and compute resources, deep learning achieves many successes in various domains. The application of deep learning on the mobile and embedded devices is taken more and more attentions, benchmarking and ranking the AI abilities of mobile and embedded devices becomes an urgent problem to be solved. Considering the model diversity and framework diversity, we propose a benchmark suite, AIoTBench, which focuses on the evaluation of the inference abilities of mobile and embedded devices. AIoTBench covers three typical heavy-weight networks: ResNet50, InceptionV3, DenseNet121, as well as three light-weight networks: SqueezeNet, MobileNetV2, MnasNet. Each network is implemented by three frameworks which are designed for mobile and embedded devices: Tensorflow Lite, Caffe2, Pytorch Mobile. To compare and rank the AI capabilities of the devices, we propose two unified metrics as the AI scores: Valid Images Per Second (VIPS) and Valid FLOPs Per Second (VOPS). Currently, we have compared and ranked 5 mobile devices using our benchmark. This list will be extended and updated soon after.
References in corpus (7)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- Benchmark Analysis of Representative Deep Neural Network Architectures
- Rigid-Motion Scattering for Texture Classification
- Fathom: Reference Workloads for Modern Deep Learning Methods
- Progressive Neural Architecture Search
- On-Device Neural Net Inference with Mobile GPUs
Cited by in corpus (6)
- Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models
- Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Quality
- HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
- HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking
- ML-EXray: Visibility into ML Deployment on the Edge
- Single-Training Collaborative Object Detectors Adaptive to Bandwidth and Computation