Benchmark Analysis of Representative Deep Neural Network Architectures

Abstract

This work presents an in-depth analysis of the majority of the deep neural networks (DNNs) proposed in the state of the art for image recognition. For each DNN multiple performance indices are observed, such as recognition accuracy, model complexity, computational complexity, memory usage, and inference time. The behavior of such performance indices and some combinations of them are analyzed and discussed. To measure the indices we experiment the use of DNNs on two different computer architectures, a workstation equipped with a NVIDIA Titan X Pascal and an embedded system based on a NVIDIA Jetson TX1 board. This experimentation allows a direct comparison between DNNs running on machines with very different computational capacity. This study is useful for researchers to have a complete view of what solutions have been explored so far and in which research directions are worth exploring in the future; and for practitioners to select the DNN architecture(s) that better fit the resource constraints of practical deployments and applications. To complete this work, all the DNNs, as well as the software used for the analysis, are available online.

Will appear in IEEE Access

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Dual Path Networks

Cited by in corpus (62)

• A Light CNN for detecting COVID-19 from CT scans of the chest
• Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
• Compute and Energy Consumption Trends in Deep Learning Inference
• High Quality Monocular Depth Estimation via Transfer Learning
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• Deep Learning for Automatic Pneumonia Detection
• Aerial Images Processing for Car Detection using Convolutional Neural Networks: Comparison between Faster R-CNN and YoloV3
• What is the State of Neural Network Pruning?
• EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices
• Distributed Reinforcement Learning for Flexible and Efficient UAV Swarm Control
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• A Transfer Learning Evaluation of Deep Neural Networks for Image Classification
• Fine-Grained Sports, Yoga, and Dance Postures Recognition: A Benchmark Analysis
• PLAM: a Posit Logarithm-Approximate Multiplier
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices
• MoCA: Memory-Centric, Adaptive Execution for Multi-Tenant Deep Neural Networks
• Energy efficiency in Edge TPU vs. embedded GPU for computer-aided medical imaging segmentation and classification
• TinyML for Ubiquitous Edge AI
• PreVIous: A Methodology for Prediction of Visual Inference Performance on IoT Devices
• Binary Neural Networks for Memory-Efficient and Effective Visual Place Recognition in Changing Environments
• Goal-Oriented Scheduling in Sensor Networks with Application Timing Awareness
• An explainable deep vision system for animal classification and detection in trail-camera images with automatic post-deployment retraining
• Mobile-URSONet: an Embeddable Neural Network for Onboard Spacecraft Pose Estimation
• Taming Visually Guided Sound Generation
• Confidential Deep Learning: Executing Proprietary Models on Untrusted Devices
• DASS: Differentiable Architecture Search for Sparse neural networks
• Single-Shot 3D Detection of Vehicles from Monocular RGB Images via Geometry Constrained Keypoints in Real-Time
• DUDES: Deep Uncertainty Distillation using Ensembles for Semantic Segmentation
• SatImNet: Structured and Harmonised Training Data for Enhanced Satellite Imagery Classification
• Intra-model Variability in COVID-19 Classification Using Chest X-ray Images
• Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles
• Efficient Attention Network: Accelerate Attention by Searching Where to Plug
• Latency-Constrained DNN Architecture Learning for Edge Systems using Zerorized Batch Normalization
• Low-Power Object Counting with Hierarchical Neural Networks
• BenchCouncil's View on Benchmarking AI and Other Emerging Workloads
• The Newspaper Navigator Dataset: Extracting And Analyzing Visual Content from 16 Million Historic Newspaper Pages in Chronicling America
• Efficient Perception, Planning, and Control Algorithm for Vision-Based Automated Vehicles
• PRGFlow: Benchmarking SWAP-Aware Unified Deep Visual Inertial Odometry
• Optimizing Prediction Serving on Low-Latency Serverless Dataflow
• Performance Prediction for Convolutional Neural Networks in Edge Devices
• Influence of image noise on crack detection performance of deep convolutional neural networks
• PACiM: A Sparsity-Centric Hybrid Compute-in-Memory Architecture via Probabilistic Approximation
• INFaaS: A Model-less and Managed Inference Serving System
• LiveChess2FEN: a Framework for Classifying Chess Pieces based on CNNs
• AIBench Training: Balanced Industry-Standard AI Training Benchmarking
• MDInference: Balancing Inference Accuracy and Latency for Mobile Applications
• Implementation of a Binary Neural Network on a Passive Array of Magnetic Tunnel Junctions
• Analysis of skin lesion images with deep learning
• Supervised dimensionality reduction by a Linear Discriminant Analysis on pre-trained CNN features
• Semi-Streaming Architecture: A New Design Paradigm for CNN Implementation on FPGAs
• AutoPruning for Deep Neural Network with Dynamic Channel Masking
• MGAS: Multi-Granularity Architecture Search for Trade-Off Between Model Effectiveness and Efficiency
• Deep Sensing of Urban Waterlogging
• A Platform to Collect, Unify, and Distribute Inertial Labeled Signals for Human Activity Recognition
• Learning to aggregate feature representations
• Generalization of Deep Convolutional Neural Networks -- A Case-study on Open-source Chest Radiographs
• 4-Connected Shift Residual Networks
• Mining Domain Knowledge: Improved Framework towards Automatically Standardizing Anatomical Structure Nomenclature in Radiotherapy
• Spatial Sampling Network for Fast Scene Understanding
• Pathology-Aware Generative Adversarial Networks for Medical Image Augmentation