Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead

Abstract

Currently, Machine Learning (ML) is becoming ubiquitous in everyday life. Deep Learning (DL) is already present in many applications ranging from computer vision for medicine to autonomous driving of modern cars as well as other sectors in security, healthcare, and finance. However, to achieve impressive performance, these algorithms employ very deep networks, requiring a significant computational power, both during the training and inference time. A single inference of a DL model may require billions of multiply-and-accumulated operations, making the DL extremely compute- and energy-hungry. In a scenario where several sophisticated algorithms need to be executed with limited energy and low latency, the need for cost-effective hardware platforms capable of implementing energy-efficient DL execution arises. This paper first introduces the key properties of two brain-inspired models like Deep Neural Network (DNN), and Spiking Neural Network (SNN), and then analyzes techniques to produce efficient and high-performance designs. This work summarizes and compares the works for four leading platforms for the execution of algorithms such as CPU, GPU, FPGA and ASIC describing the main solutions of the state-of-the-art, giving much prominence to the last two solutions since they offer greater design flexibility and bear the potential of high energy-efficiency, especially for the inference process. In addition to hardware solutions, this paper discusses some of the important security issues that these DNN and SNN models may have during their execution, and offers a comprehensive section on benchmarking, explaining how to assess the quality of different networks and hardware systems designed for them.

Accepted for publication in IEEE Access

References in corpus (64)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Explaining and Harnessing Adversarial Examples
• Intriguing properties of neural networks
• ADADELTA: An Adaptive Learning Rate Method
• A Survey of the Recent Architectures of Deep Convolutional Neural Networks
• FitNets: Hints for Thin Deep Nets
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• A Survey of Deep Learning Techniques for Autonomous Driving
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Do Deep Nets Really Need to be Deep?
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• cuDNN: Efficient Primitives for Deep Learning
• Compressing Deep Convolutional Networks using Vector Quantization
• Binarized Neural Networks
• Ternary Weight Networks
• Stealing Machine Learning Models via Prediction APIs
• Certified Adversarial Robustness via Randomized Smoothing
• Fast is better than free: Revisiting adversarial training
• Accurate deep neural network inference using computational phase-change memory
• SLAYER: Spike Layer Error Reassignment in Time
• ResNeSt: Split-Attention Networks
• A Survey of Neuromorphic Computing and Neural Networks in Hardware
• Convolutional Neural Networks using Logarithmic Data Representation
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• A Survey on Methods and Theories of Quantized Neural Networks
• SCALE-Sim: Systolic CNN Accelerator Simulator
• Origami: A 803 GOp/s/W Convolutional Network Accelerator
• XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference
• ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining
• Neuromorphic Hardware In The Loop: Training a Deep Spiking Network on the BrainScaleS Wafer-Scale System
• Robust Machine Learning Systems: Challenges, Current Trends, Perspectives, and the Road Ahead
• Guessing Smart: Biased Sampling for Efficient Black-Box Adversarial Attacks
• Optimizing CNN Model Inference on CPUs
• FSpiNN: An Optimization Framework for Memory- and Energy-Efficient Spiking Neural Networks
• On Neural Architecture Search for Resource-Constrained Hardware Platforms
• Augment your batch: better training with larger batches
• A Systematic Approach to Blocking Convolutional Neural Networks
• Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation
• NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks
• AxTrain: Hardware-Oriented Neural Network Training for Approximate Inference
• Is Spiking Secure? A Comparative Study on the Security Vulnerabilities of Spiking and Deep Neural Networks
• Security for Machine Learning-based Systems: Attacks and Challenges during Training and Inference
• QuSecNets: Quantization-based Defense Mechanism for Securing Deep Neural Network against Adversarial Attacks
• CapsAcc: An Efficient Hardware Accelerator for CapsuleNets with Data Reuse
• NeuroAttack: Undermining Spiking Neural Networks Security through Externally Triggered Bit-Flips
• Debugging Machine Learning Tasks
• TrISec: Training Data-Unaware Imperceptible Security Attacks on Deep Neural Networks
• Q-CapsNets: A Specialized Framework for Quantizing Capsule Networks
• CapsAttacks: Robust and Imperceptible Adversarial Attacks on Capsule Networks
• DESCNet: Developing Efficient Scratchpad Memories for Capsule Network Hardware
• ReD-CaNe: A Systematic Methodology for Resilience Analysis and Design of Capsule Networks under Approximations
• MPNA: A Massively-Parallel Neural Array Accelerator with Dataflow Optimization for Convolutional Neural Networks
• RED-Attack: Resource Efficient Decision based Attack for Machine Learning
• Closing the Accuracy Gap in an Event-Based Visual Recognition Task
• CapStore: Energy-Efficient Design and Management of the On-Chip Memory for CapsuleNet Inference Accelerators
• A Low-Power Accelerator for Deep Neural Networks with Enlarged Near-Zero Sparsity

Cited by in corpus (17)

• Structured Pruning for Deep Convolutional Neural Networks: A survey
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• Towards Energy-Efficient and Secure Edge AI: A Cross-Layer Framework
• Securing Deep Spiking Neural Networks against Adversarial Attacks through Inherent Structural Parameters
• ROMANet: Fine-Grained Reuse-Driven Off-Chip Memory Access Management and Data Organization for Deep Neural Network Accelerators
• Exposing Reliability Degradation and Mitigation in Approximate DNNs under Permanent Faults
• Optimizing Dense Feed-Forward Neural Networks
• Neural Network Methods for Radiation Detectors and Imaging
• A Homomorphic Encryption Framework for Privacy-Preserving Spiking Neural Networks
• RoHNAS: A Neural Architecture Search Framework with Conjoint Optimization for Adversarial Robustness and Hardware Efficiency of Convolutional and Capsule Networks
• FastCaps: A Design Methodology for Accelerating Capsule Network on Field Programmable Gate Arrays
• Implementation of a Binary Neural Network on a Passive Array of Magnetic Tunnel Junctions
• "I see models being a whole other thing": An Empirical Study of Pre-Trained Model Naming Conventions and A Tool for Enhancing Naming Consistency
• Improving deep neural network performance through sampling
• Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems
• Fast Exploration of the Impact of Precision Reduction on Spiking Neural Networks
• A deep learning approach to the texture optimization problem for friction control in lubricated contacts