EIE: Efficient Inference Engine on Compressed Deep Neural Network

Abstract

State-of-the-art deep neural networks (DNNs) have hundreds of millions of connections and are both computationally and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources and power budgets. While custom hardware helps the computation, fetching weights from DRAM is two orders of magnitude more expensive than ALU operations, and dominates the required power. Previously proposed 'Deep Compression' makes it possible to fit large DNNs (AlexNet and VGGNet) fully in on-chip SRAM. This compression is achieved by pruning the redundant connections and having multiple connections share the same weight. We propose an energy efficient inference engine (EIE) that performs inference on this compressed network model and accelerates the resulting sparse matrix-vector multiplication with weight sharing. Going from DRAM to SRAM gives EIE 120x energy saving; Exploiting sparsity saves 10x; Weight sharing gives 8x; Skipping zero activations from ReLU saves another 3x. Evaluated on nine DNN benchmarks, EIE is 189x and 13x faster when compared to CPU and GPU implementations of the same DNN without compression. EIE has a processing power of 102GOPS/s working directly on a compressed network, corresponding to 3TOPS/s on an uncompressed network, and processes FC layers of AlexNet at 1.88x10^4 frames/sec with a power dissipation of only 600mW. It is 24,000x and 3,400x more energy efficient than a CPU and GPU respectively. Compared with DaDianNao, EIE has 2.9x, 19x and 3x better throughput, energy efficiency and area efficiency.

External Links: TheNextPlatform: http://goo.gl/f7qX0L ; O'Reilly: https://goo.gl/Id1HNT ; Hacker News: https://goo.gl/KM72SV ; Embedded-vision: http://goo.gl/joQNg8 ; Talk at NVIDIA GTC'16: http://goo.gl/6wJYvn ; Talk at Embedded Vision Summit: https://goo.gl/7abFNe ; Talk at Stanford University: https://goo.gl/6lwuer. Published as a conference paper in ISCA 2016

Cited by in corpus (40)

• Embedding Watermarks into Deep Neural Networks
• FPGA-based Accelerators of Deep Learning Networks for Learning and Classification: A Review
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning
• Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators
• Fathom: Reference Workloads for Modern Deep Learning Methods
• Survey of Machine Learning Accelerators
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Recurrent Neural Networks: An Embedded Computing Perspective
• TinyLSTMs: Efficient Neural Speech Enhancement for Hearing Aids
• Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Search
• Accelerator-Aware Pruning for Convolutional Neural Networks
• A Systematic Review on Model Watermarking for Neural Networks
• Balanced Sparsity for Efficient DNN Inference on GPU
• VIBNN: Hardware Acceleration of Bayesian Neural Networks
• Robust Ultra-wideband Range Error Mitigation with Deep Learning at the Edge
• Cluster Pruning: An Efficient Filter Pruning Method for Edge AI Vision Applications
• A Stochastic-Computing based Deep Learning Framework using Adiabatic Quantum-Flux-Parametron SuperconductingTechnology
• Capstan: A Vector RDA for Sparsity
• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks
• Understanding Real-world Threats to Deep Learning Models in Android Apps
• Dynamically Sacrificing Accuracy for Reduced Computation: Cascaded Inference Based on Softmax Confidence
• Throughput Optimizations for FPGA-based Deep Neural Network Inference
• SPRING: A Sparsity-Aware Reduced-Precision Monolithic 3D CNN Accelerator Architecture for Training and Inference
• Task dependent Deep LDA pruning of neural networks
• Lightweight Residual Densely Connected Convolutional Neural Network
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Group channel pruning and spatial attention distilling for object detection
• DESCNet: Developing Efficient Scratchpad Memories for Capsule Network Hardware
• SparseTrain:Leveraging Dynamic Sparsity in Training DNNs on General-Purpose SIMD Processors
• Pack and Detect: Fast Object Detection in Videos Using Region-of-Interest Packing
• Tensor Slicing and Optimization for Multicore NPUs
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• Approximate Computing and the Efficient Machine Learning Expedition
• A Gradient-Interleaved Scheduler for Energy-Efficient Backpropagation for Training Neural Networks
• COPS: Controlled Pruning Before Training Starts
• A 1.6-mW Sparse Deep Learning Accelerator for Speech Separation
• Programming with Neural Surrogates of Programs
• GrateTile: Efficient Sparse Tensor Tiling for CNN Processing