Approximations in Deep Learning

Abstract

The design and implementation of Deep Learning (DL) models is currently receiving a lot of attention from both industrials and academics. However, the computational workload associated with DL is often out of reach for low-power embedded devices and is still costly when run on datacenters. By relaxing the need for fully precise operations, Approximate Computing (AxC) substantially improves performance and energy efficiency. DL is extremely relevant in this context, since playing with the accuracy needed to do adequate computations will significantly enhance performance, while keeping the quality of results in a user-constrained range. This chapter will explore how AxC can improve the performance and energy efficiency of hardware accelerators in DL applications during inference and training.

arXiv commentApproximate Computing Techniques - From Component- to Application-Level, pp.467-512, 2022, 978-3-030-94704-0

References in corpus (18)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Fixed Point Quantization of Deep Convolutional Networks
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Accurate deep neural network inference using computational phase-change memory
• SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
• Towards Accurate Binary Convolutional Neural Network
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• Progressive Weight Pruning of Deep Neural Networks using ADMM
• Learning Sparse Low-Precision Neural Networks With Learnable Regularization