Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey

Abstract

Deep Neural Networks (DNNs) are very popular because of their high performance in various cognitive tasks in Machine Learning (ML). Recent advancements in DNNs have brought beyond human accuracy in many tasks, but at the cost of high computational complexity. To enable efficient execution of DNN inference, more and more research works, therefore, exploit the inherent error resilience of DNNs and employ Approximate Computing (AC) principles to address the elevated energy demands of DNN accelerators. This article provides a comprehensive survey and analysis of hardware approximation techniques for DNN accelerators. First, we analyze the state of the art and by identifying approximation families, we cluster the respective works with respect to the approximation type. Next, we analyze the complexity of the performed evaluations (with respect to the dataset and DNN size) to assess the efficiency, the potential, and limitations of approximate DNN accelerators. Moreover, a broad discussion is provided, regarding error metrics that are more suitable for designing approximate units for DNN accelerators as well as accuracy recovery approaches that are tailored to DNN inference. Finally, we present how Approximate Computing for DNN accelerators can go beyond energy efficiency and address reliability and security issues, as well.

arXiv commentThis paper has been accepted by ACM Computing Surveys (CSUR), 2022

References in corpus (27)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Deep Learning with Limited Numerical Precision
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• Trained Ternary Quantization
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference
• ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining
• SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
• Robust Machine Learning Systems: Challenges, Current Trends, Perspectives, and the Road Ahead
• How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review
• Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets
• AddNet: Deep Neural Networks Using FPGA-Optimized Multipliers
• Discretization based Solutions for Secure Machine Learning against Adversarial Attacks
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Defensive Approximation: Securing CNNs using Approximate Computing
• A Survey of Quantization Methods for Efficient Neural Network Inference
• TFApprox: Towards a Fast Emulation of DNN Approximate Hardware Accelerators on GPU
• Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
• Automated Circuit Approximation Method Driven by Data Distribution

Cited by in corpus (15)

• Federated Learning for Computationally-Constrained Heterogeneous Devices: A Survey
• AdaPT: Fast Emulation of Approximate DNN Accelerators in PyTorch
• Approximate Computing Survey, Part I: Terminology and Software & Hardware Approximation Techniques
• Exposing Reliability Degradation and Mitigation in Approximate DNNs under Permanent Faults
• Mixed-precision Neural Networks on RISC-V Cores: ISA extensions for Multi-Pumped Soft SIMD Operations
• Approximate Computing Survey, Part II: Application-Specific & Architectural Approximation Techniques and Applications
• Model-to-Circuit Cross-Approximation For Printed Machine Learning Classifiers
• Bespoke Approximation of Multiplication-Accumulation and Activation Targeting Printed Multilayer Perceptrons
• X-Rel: Energy-Efficient and Low-Overhead Approximate Reliability Framework for Error-Tolerant Applications Deployed in Critical Systems
• HOAA: Hybrid Overestimating Approximate Adder for Enhanced Performance Processing Engine
• Embedding Hardware Approximations in Discrete Genetic-based Training for Printed MLPs
• Energy-Aware Heterogeneous Federated Learning via Approximate DNN Accelerators
• ApproxDARTS: Differentiable Neural Architecture Search with Approximate Multipliers
• TransAxx: Efficient Transformers with Approximate Computing
• Exploring Quantization and Mapping Synergy in Hardware-Aware Deep Neural Network Accelerators