Dynamic Neural Networks: A Survey

Abstract

Dynamic neural network is an emerging research topic in deep learning. Compared to static models which have fixed computational graphs and parameters at the inference stage, dynamic networks can adapt their structures or parameters to different inputs, leading to notable advantages in terms of accuracy, computational efficiency, adaptiveness, etc. In this survey, we comprehensively review this rapidly developing area by dividing dynamic networks into three main categories: 1) instance-wise dynamic models that process each instance with data-dependent architectures or parameters; 2) spatial-wise dynamic networks that conduct adaptive computation with respect to different spatial locations of image data and 3) temporal-wise dynamic models that perform adaptive inference along the temporal dimension for sequential data such as videos and texts. The important research problems of dynamic networks, e.g., architecture design, decision making scheme, optimization technique and applications, are reviewed systematically. Finally, we discuss the open problems in this field together with interesting future research directions.

References in corpus (37)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Neural Architecture Search with Reinforcement Learning
• Transformers in Vision: A Survey
• On Calibration of Modern Neural Networks
• Recurrent Models of Visual Attention
• Session-based Social Recommendation via Dynamic Graph Attention Networks
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• Modulating early visual processing by language
• Learning Efficient Convolutional Networks through Network Slimming
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• Distilling a Neural Network Into a Soft Decision Tree
• Deep Multimodal Fusion by Channel Exchanging
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
• Dynamic Sampling Networks for Efficient Action Recognition in Videos
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• Meta-SR: A Magnification-Arbitrary Network for Super-Resolution
• Discrete Autoencoders for Sequence Models
• Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation
• Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference
• Video Frame Interpolation via Adaptive Convolution
• LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognition
• Variable Computation in Recurrent Neural Networks
• Scale-Aware Face Detection
• Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning
• Pixel-Adaptive Convolutional Neural Networks
• Changing Model Behavior at Test-Time Using Reinforcement Learning
• ELF: An Early-Exiting Framework for Long-Tailed Classification
• Neural Speed Reading with Structural-Jump-LSTM
• Anisotropic Convolutional Networks for 3D Semantic Scene Completion
• TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning
• Adaptive Feeding: Achieving Fast and Accurate Detections by Adaptively Combining Object Detectors
• S2DNAS:Transforming Static CNN Model for Dynamic Inference via Neural Architecture Search
• Adaptive Focus for Efficient Video Recognition

Cited by in corpus (18)

• Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• SimMIM: A Simple Framework for Masked Image Modeling
• Set-to-Sequence Methods in Machine Learning: a Review
• Adaptive Focus for Efficient Video Recognition
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• Dynamic Image Restoration and Fusion Based on Dynamic Degradation
• Federated Dynamic Neural Network for Deep MIMO Detection
• Dynamic Slimmable Network
• Dynamic Slimmable Denoising Network
• CondenseNet V2: Sparse Feature Reactivation for Deep Networks
• Temporal Dynamic Convolutional Neural Network for Text-Independent Speaker Verification and Phonemetic Analysis
• A Survey on Green Deep Learning
• DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework
• Partial to Whole Knowledge Distillation: Progressive Distilling Decomposed Knowledge Boosts Student Better
• Dynamic Parameterized Network for CTR Prediction