Dynamic Neural Networks: A Survey

Abstract

Dynamic neural network is an emerging research topic in deep learning. Compared to static models which have fixed computational graphs and parameters at the inference stage, dynamic networks can adapt their structures or parameters to different inputs, leading to notable advantages in terms of accuracy, computational efficiency, adaptiveness, etc. In this survey, we comprehensively review this rapidly developing area by dividing dynamic networks into three main categories: 1) instance-wise dynamic models that process each instance with data-dependent architectures or parameters; 2) spatial-wise dynamic networks that conduct adaptive computation with respect to different spatial locations of image data and 3) temporal-wise dynamic models that perform adaptive inference along the temporal dimension for sequential data such as videos and texts. The important research problems of dynamic networks, e.g., architecture design, decision making scheme, optimization technique and applications, are reviewed systematically. Finally, we discuss the open problems in this field together with interesting future research directions.

References in corpus (78)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Neural Architecture Search with Reinforcement Learning
• Transformers in Vision: A Survey
• Categorical Reparameterization with Gumbel-Softmax
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• On Calibration of Modern Neural Networks
• DARTS: Differentiable Architecture Search
• Recurrent Models of Visual Attention
• Binarized Neural Networks
• Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
• Session-based Social Recommendation via Dynamic Graph Attention Networks
• Universal Transformers
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Adaptive Computation Time for Recurrent Neural Networks
• Residual Attention Network for Image Classification
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• Modulating early visual processing by language
• ReasoNet: Learning to Stop Reading in Machine Comprehension
• Learning Efficient Convolutional Networks through Network Slimming
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Distilling a Neural Network Into a Soft Decision Tree
• Semantic Image Synthesis with Spatially-Adaptive Normalization
• CondConv: Conditionally Parameterized Convolutions for Efficient Inference
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Conditional Computation in Neural Networks for faster models
• Channel Gating Neural Networks
• Compact Generalized Non-local Network
• Deep Multimodal Fusion by Channel Exchanging
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
• Dual Dynamic Inference: Enabling More Efficient, Adaptive and Controllable Deep Inference
• MetaAnchor: Learning to Detect Objects with Customized Anchors
• Dynamic Sampling Networks for Efficient Action Recognition in Videos
• Neural Speed Reading via Skim-RNN
• Implicit Semantic Data Augmentation for Deep Networks
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Deciding How to Decide: Dynamic Routing in Artificial Neural Networks
• Deep Mixture of Experts via Shallow Embedding
• Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs
• Batch-Shaping for Learning Conditional Channel Gated Networks
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• Meta-SR: A Magnification-Arbitrary Network for Super-Resolution
• CARAFE: Content-Aware ReAssembly of FEatures
• More is Less: A More Complicated Network with Less Inference Complexity
• Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation
• Discrete Autoencoders for Sequence Models
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• Video Frame Interpolation via Adaptive Convolution
• LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition
• Variable Computation in Recurrent Neural Networks
• Scale-Aware Face Detection
• Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning
• IamNN: Iterative and Adaptive Mobile Neural Network for Efficient Image Classification
• Online Video Deblurring via Dynamic Temporal Blending Network
• Focused Hierarchical RNNs for Conditional Sequence Processing
• Spatio-Temporal Filter Adaptive Network for Video Deblurring
• Not All Pixels Are Equal: Difficulty-aware Semantic Segmentation via Deep Layer Cascade
• Adaptive Neural Trees
• Resolution Adaptive Networks for Efficient Inference
• Question-Guided Hybrid Convolution for Visual Question Answering
• Pixel-Adaptive Convolutional Neural Networks
• Changing Model Behavior at Test-Time Using Reinforcement Learning
• ELF: An Early-Exiting Framework for Long-Tailed Classification
• Neural Speed Reading with Structural-Jump-LSTM
• Anisotropic Convolutional Networks for 3D Semantic Scene Completion
• SBNet: Sparse Blocks Network for Fast Inference
• Learning Dynamic Routing for Semantic Segmentation
• TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning
• Adaptive Feeding: Achieving Fast and Accurate Detections by Adaptively Combining Object Detectors
• S2DNAS:Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search
• Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks
• Adaptive Focus for Efficient Video Recognition

Cited by in corpus (18)

• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• SimMIM: A Simple Framework for Masked Image Modeling
• Set-to-Sequence Methods in Machine Learning: a Review
• Adaptive Focus for Efficient Video Recognition
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• Dynamic Image Restoration and Fusion Based on Dynamic Degradation
• Federated Dynamic Neural Network for Deep MIMO Detection
• Dynamic Slimmable Network
• Dynamic Slimmable Denoising Network
• CondenseNet V2: Sparse Feature Reactivation for Deep Networks
• Temporal Dynamic Convolutional Neural Network for Text-Independent Speaker Verification and Phonemetic Analysis
• A Survey on Green Deep Learning
• DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework
• Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better
• Dynamic Parameterized Network for CTR Prediction