Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions

Abstract

Visual recognition is currently one of the most important and active research areas in computer vision, pattern recognition, and even the general field of artificial intelligence. It has great fundamental importance and strong industrial needs. Deep neural networks (DNNs) have largely boosted their performances on many concrete tasks, with the help of large amounts of training data and new powerful computation resources. Though recognition accuracy is usually the first concern for new progresses, efficiency is actually rather important and sometimes critical for both academic research and industrial applications. Moreover, insightful views on the opportunities and challenges of efficiency are also highly required for the entire community. While general surveys on the efficiency issue of DNNs have been done from various perspectives, as far as we are aware, scarcely any of them focused on visual recognition systematically, and thus it is unclear which progresses are applicable to it and what else should be concerned. In this paper, we present the review of the recent advances with our suggestions on the new possible directions towards improving the efficiency of DNN-related visual recognition approaches. We investigate not only from the model but also the data point of view (which is not the case in existing surveys), and focus on three most studied data types (images, videos and points). This paper attempts to provide a systematic summary via a comprehensive survey which can serve as a valuable reference and inspire both researchers and practitioners who work on visual recognition problems.

References in corpus (149)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• LSTM: A Search Space Odyssey
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• ShapeNet: An Information-Rich 3D Model Repository
• Event-based Vision: A Survey
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Neural Architecture Search: A Survey
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Multi-Scale Context Aggregation by Dilated Convolutions
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• Tensor Decompositions for Signal Processing Applications From Two-way to Multiway Component Analysis
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Variational image compression with a scale hyperprior
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• cuDNN: Efficient Primitives for Deep Learning
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Mixed Precision Training
• Rethinking the Value of Network Pruning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Trained Ternary Quantization
• Predicting Parameters in Deep Learning
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Tensorizing Neural Networks
• Compressing Neural Networks with the Hashing Trick
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Trunk-Branch Ensemble Convolutional Neural Networks for Video-based Face Recognition
• Deformable Convolutional Networks
• Designing Neural Network Architectures using Reinforcement Learning
• Fast-SCNN: Fast Semantic Segmentation Network
• Joint Autoregressive and Hierarchical Priors for Learned Image Compression
• Algorithms for entanglement renormalization
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Quasi-Recurrent Neural Networks
• Training and Inference with Integers in Deep Neural Networks
• Efficient Architecture Search by Network Transformation
• Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
• Tensor Ring Decomposition
• PEPS as ground states: degeneracy and topology
• Grid Long Short-Term Memory
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Model compression via distillation and quantization
• NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision
• Learned Point Cloud Geometry Compression
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Slimmable Neural Networks
• Hierarchical Representations for Efficient Architecture Search
• Video Compression With Rate-Distortion Autoencoders
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Dynamic Channel Pruning: Feature Boosting and Suppression
• ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time
• Scalable Methods for 8-bit Training of Neural Networks
• Learning Convolutional Transforms for Lossy Point Cloud Geometry Compression
• WRPN: Wide Reduced-Precision Networks
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
• Sequential Neural Models with Stochastic Layers
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Path-Level Network Transformation for Efficient Architecture Search
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
• Neural Network Renormalization Group
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• Adaptive Neural Networks for Efficient Inference
• Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
• On the Expressive Power of Deep Learning: A Tensor Analysis
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications
• Two-Stream 3D Convolutional Neural Network for Skeleton-Based Action Recognition
• Compressing Recurrent Neural Network with Tensor Train
• Architectural Complexity Measures of Recurrent Neural Networks
• Towards Image Understanding from Deep Compression without Decoding
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Ultimate tensorization: compressing convolutional and FC layers alike
• Once-for-All: Train One Network and Specialize it for Efficient Deployment
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Training Quantized Nets: A Deeper Understanding
• Tensor Regression Networks
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• The unreasonable effectiveness of the forget gate
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• A novel channel pruning method for deep neural network compression
• Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation
• Deep AutoEncoder-based Lossy Geometry Compression for Point Clouds
• Tensor-Train Recurrent Neural Networks for Video Classification
• A literature survey of low-rank tensor approximation techniques
• VecQ: Minimal Loss DNN Model Compression With Vectorized Weight Quantization
• Crossbar-aware neural network pruning
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• From Nodes to Networks: Evolving Recurrent Neural Networks
• Hybrid Tensor Decomposition in Neural Network Compression
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• Are All Training Examples Created Equal? An Empirical Study
• Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
• From Hashing to CNNs: Training BinaryWeight Networks via Hashing
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• Progressive Weight Pruning of Deep Neural Networks using ADMM
• Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks
• AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference
• Dual Path Networks
• Sharing Residual Units Through Collective Tensor Factorization in Deep Neural Networks
• Make Skeleton-based Action Recognition Model Smaller, Faster and Better
• Kronecker CP Decomposition with Fast Multiplication for Compressing RNNs
• Learning Sparse Low-Precision Neural Networks With Learnable Regularization
• Hybrid Pruning: Thinner Sparse Networks for Fast Inference on Edge Devices
• Compressing 3DCNNs Based on Tensor Train Decomposition
• Compressing the Input for CNNs with the First-Order Scattering Transform
• Synaptic Strength For Convolutional Neural Network
• Convolutional Rectifier Networks as Generalized Tensor Decompositions
• Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decomposition
• AutoQ: Automated Kernel-Wise Neural Network Quantization
• Regularized Computation of Approximate Pseudoinverse of Large Matrices Using Low-Rank Tensor Train Decompositions
• Learning Efficient Tensor Representations with Ring Structure Networks
• Investigating gated recurrent neural networks for speech synthesis
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• Semantic Redundancies in Image-Classification Datasets: The 10% You Don't Need
• Grow and Prune Compact, Fast, and Accurate LSTMs
• Adaptive Quantization for Deep Neural Network
• Towards Understanding the Transferability of Deep Representations
• Optimizing Video Object Detection via a Scale-Time Lattice
• Tensor Contraction Layers for Parsimonious Deep Nets
• Per-Tensor Fixed-Point Quantization of the Back-Propagation Algorithm
• BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights
• Training Compact Neural Networks with Binary Weights and Low Precision Activations
• DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework
• Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training
• Singular-Value-Decomposition Analysis of Associative Memory in a Neural Network
• End-to-end Learning of a Convolutional Neural Network via Deep Tensor Decomposition
• Faster and Accurate Classification for JPEG2000 Compressed Images in Networked Applications
• Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
• CP-decomposition with Tensor Power Method for Convolutional Neural Networks Compression
• URNet : User-Resizable Residual Networks with Conditional Gating Module
• Recurrent Convolution for Compact and Cost-Adjustable Neural Networks: An Empirical Study