Deep ReLU Networks Preserve Expected Length
arXiv:2102.10492
Abstract
Assessing the complexity of functions computed by a neural network helps us understand how the network will learn and generalize. One natural measure of complexity is how the network distorts length - if the network takes a unit-length curve as input, what is the length of the resulting curve of outputs? It has been widely believed that this length grows exponentially in network depth. We prove that in fact this is not the case: the expected length distortion does not grow with depth, and indeed shrinks slightly, for ReLU networks with standard random initialization. We also generalize this result by proving upper bounds both for higher moments of the length distortion and for the distortion of higher-dimensional volumes. These theoretical results are corroborated by our experiments.
arXiv comment18 pages, 4 figures
References in corpus (14)
- Do Deep Nets Really Need to be Deep?
- On the Number of Linear Regions of Deep Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Exponential expressivity in deep neural networks through transient chaos
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Spectrally-normalized margin bounds for neural networks
- Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?
- Fantastic Generalization Measures and Where to Find Them
- Representation Benefits of Deep Feedforward Networks
- The power of deeper networks for expressing natural functions
- Deep ReLU Networks Have Surprisingly Few Activation Patterns
- Complexity of Linear Regions in Deep Networks
- Products of Many Large Random Matrices and Gradients in Deep Neural Networks
- Depth Separation for Neural Networks