Finite-time Lyapunov exponents of deep neural networks
arXiv:2306.12548 · doi:10.1103/PhysRevLett.132.057301
Abstract
We compute how small input perturbations affect the output of deep neural networks, exploring an analogy between deep networks and dynamical systems, where the growth or decay of local perturbations is characterised by finite-time Lyapunov exponents. We show that the maximal exponent forms geometrical structures in input space, akin to coherent structures in dynamical systems. Ridges of large positive exponents divide input space into different regions that the network associates with different classes. These ridges visualise the geometry that deep networks construct in input space, shedding light on the fundamental mechanisms underlying their learning capabilities.
arXiv comment6 pages, 4 figures
References in corpus (15)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Visualizing the geometry of state space in plane Couette flow
- Exponential expressivity in deep neural networks through transient chaos
- A Survey of Uncertainty in Deep Neural Networks
- Edge States in the Climate System: Exploring Global Instabilities and Critical Transitions
- Alignment of vorticity and rods with Lagrangian fluid stretching in turbulence
- Machine learning with neural networks
- Predictability of large-scale atmospheric motions: Lyapunov exponents and error dynamics
- Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
- A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
- Geometric compression of invariant manifolds in neural nets
- Poincare Indices of Rheoscopic Visualisations
- The edge as a Lagrangian Coherent Structure in a high-dimensional state space
- On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
- Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity