Model Complexity of Deep Learning: A Survey
arXiv:2103.05127
Abstract
Model complexity is a fundamental problem in deep learning. In this paper we conduct a systematic overview of the latest studies on model complexity in deep learning. Model complexity of deep learning can be categorized into expressive capacity and effective model complexity. We review the existing studies on those two categories along four important factors, including model framework, model size, optimization process and data complexity. We also discuss the applications of deep learning model complexity including understanding model generalization, model optimization, and model selection and design. We conclude by proposing several interesting future directions.
References in corpus (19)
- Distilling the Knowledge in a Neural Network
- Neural Architecture Search with Reinforcement Learning
- Do Deep Nets Really Need to be Deep?
- DARTS: Differentiable Architecture Search
- Activation Functions: Comparison of trends in Practice and Research for Deep Learning
- On the Number of Linear Regions of Deep Neural Networks
- Exponential expressivity in deep neural networks through transient chaos
- Sensitivity and Generalization in Neural Networks: an Empirical Study
- Quantum computational finance: Monte Carlo pricing of financial derivatives
- Optimization for deep learning: theory and algorithms
- Approximating Continuous Functions by ReLU Nets of Minimal Width
- Rademacher Complexity for Adversarially Robust Generalization
- Fisher-Rao Metric, Geometry, and Complexity of Neural Networks
- Implicit Regularization in Deep Learning
- Norm-Based Capacity Control in Neural Networks
- Complexity of Linear Regions in Deep Networks
- On the Selection of Initialization and Activation Function for Deep Neural Networks
- Learning with tree tensor networks: complexity estimates and model selection
- On generic and maximal k-ranks of binary forms