Exponential expressivity in deep neural networks through transient chaos

Abstract

We combine Riemannian geometry with the mean field theory of high dimensional chaos to study the nature of signal propagation in generic, deep neural networks with random weights. Our results reveal an order-to-chaos expressivity phase transition, with networks in the chaotic phase computing nonlinear functions whose global curvature grows exponentially with depth but not width. We prove this generic class of deep random functions cannot be efficiently computed by any shallow network, going beyond prior work restricted to the analysis of single functions. Moreover, we formalize and quantitatively demonstrate the long conjectured idea that deep networks can disentangle highly curved manifolds in input space into flat manifolds in hidden space. Our theoretical analysis of the expressive power of deep networks broadly applies to arbitrary nonlinearities, and provides a quantitative underpinning for previously abstract notions about the geometry of deep functions.

Fixed equation references

References in corpus (5)

• Playing Atari with Deep Reinforcement Learning
• On the Number of Linear Regions of Deep Neural Networks
• Deep Knowledge Tracing
• The Power of Depth for Feedforward Neural Networks
• Representation Benefits of Deep Feedforward Networks

Cited by in corpus (109)

• A Closer Look at Memorization in Deep Networks
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• The Principles of Deep Learning Theory
• Superconducting optoelectronic circuits for neuromorphic computing
• Deep Residual Learning for Compressed Sensing CT Reconstruction via Persistent Homology Analysis
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• ReZero is All You Need: Fast Convergence at Large Depth
• Learning ReLU Networks on Linearly Separable Data: Algorithm, Optimality, and Generalization
• The Modern Mathematics of Deep Learning
• On the Convergence Rate of Training Recurrent Neural Networks
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Review: Deep Learning in Electron Microscopy
• Biologically inspired protection of deep networks from adversarial attacks
• Neural-net-induced Gaussian process regression for function approximation and PDE solution
• Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
• Reformulation of the No-Free-Lunch Theorem for Entangled Data Sets
• A Mean Field Theory of Batch Normalization
• Stiffness: A New Perspective on Generalization in Neural Networks
• Tensor Programs II: Neural Tangent Kernel for Any Architecture
• How to Start Training: The Effect of Initialization and Architecture
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
• Universality and individuality in neural dynamics across large populations of recurrent networks
• A Fine-Grained Spectral Perspective on Neural Networks
• Explicit Regularisation in Gaussian Noise Injections
• Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks
• A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks
• Classification regions of deep neural networks
• Physical deep learning based on optimal control of dynamical systems
• Mean-field theory of graph neural networks in graph partitioning
• Automated Architecture Design for Deep Neural Networks
• Empirical Studies on the Properties of Linear Regions in Deep Neural Networks
• On the Number of Linear Regions of Convolutional Neural Networks
• Deep Transformers with Latent Depth
• Exploring the Function Space of Deep-Learning Machines
• Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space
• Finite-time Lyapunov exponents of deep neural networks
• Feature Learning in Infinite-Width Neural Networks
• Wide and Deep Neural Networks Achieve Optimality for Classification
• A Signal Propagation Perspective for Pruning Neural Networks at Initialization
• On Random Matrices Arising in Deep Neural Networks. Gaussian Case
• Tensor Programs III: Neural Matrix Laws
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• Do Neural Nets Learn Statistical Laws behind Natural Language?
• The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
• Robust Pruning at Initialization
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• A Correspondence Between Random Neural Networks and Statistical Field Theory
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• Neural networks are a priori biased towards Boolean functions with low entropy
• Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network
• On the expected behaviour of noise regularised deep neural networks as Gaussian processes
• How deep learning works --The geometry of deep learning
• Order and Chaos: NTK views on DNN Normalization, Checkerboard and Boundary Artifacts
• Invariance of Weight Distributions in Rectified MLPs
• Pathological spectra of the Fisher information metric and its variants in deep neural networks
• Variational Implicit Processes
• Is SGD a Bayesian sampler? Well, almost
• The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
• On neural network kernels and the storage capacity problem
• Analysis on the Nonlinear Dynamics of Deep Neural Networks: Topological Entropy and Chaos
• Space of Functions Computed by Deep-Layered Machines
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• Deep ReLU Networks Preserve Expected Length
• The Nonlinearity Coefficient - Predicting Generalization in Deep Neural Networks
• Neural Anisotropy Directions
• Deep Tensor Convolution on Multicores
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• Initialization of ReLUs for Dynamical Isometry
• Energy-information trade-off makes the cortical critical power law the optimal coding
• Model Complexity of Deep Learning: A Survey
• Wide Neural Networks with Bottlenecks are Deep Gaussian Processes
• Optimal Function Approximation with Relu Neural Networks
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Fast learning rate of deep learning via a kernel perspective
• Universal scaling laws in the gradient descent training of neural networks
• Critical initialisation in continuous approximations of binary neural networks
• Neural Network Predicts Ion Concentration Profiles under Nanoconfinement
• A Deep Conditioning Treatment of Neural Networks
• Robust Adversarial Learning via Sparsifying Front Ends
• Exploring the Uncertainty Properties of Neural Networks' Implicit Priors in the Infinite-Width Limit
• Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective
• Mean Shift Rejection: Training Deep Neural Networks Without Minibatch Statistics or Normalization
• Eigenvalue Distribution of Large Random Matrices Arising in Deep Neural Networks: Orthogonal Case
• Better Depth-Width Trade-offs for Neural Networks through the lens of Dynamical Systems
• Collective evolution of weights in wide neural networks
• Deep Learning Works in Practice. But Does it Work in Theory?
• Mean field theory for deep dropout networks: digging up gradient backpropagation deeply
• Adversarial Robustness Guarantees for Random Deep Neural Networks
• Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
• Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum
• Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence
• Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?
• Depth separation beyond radial functions
• Initializing ReLU networks in an expressive subspace of weights
• DNN-Based Topology Optimisation: Spatial Invariance and Neural Tangent Kernel
• Some Geometrical and Topological Properties of DNNs' Decision Boundaries
• Sample Variance Decay in Randomly Initialized ReLU Networks
• Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
• Towards a General Theory of Infinite-Width Limits of Neural Classifiers
• Mean Field Theory of Activation Functions in Deep Neural Networks
• Fast generalization error bound of deep learning without scale invariance of activation functions
• Almost Sure Asymptotic Freeness of Neural Network Jacobian with Orthogonal Weights
• A Johnson--Lindenstrauss Framework for Randomly Initialized CNNs
• Training BatchNorm Only in Neural Architecture Search and Beyond
• On the security relevance of weights in deep learning
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks