Finite-time Lyapunov exponents of deep neural networks

Abstract

We compute how small input perturbations affect the output of deep neural networks, exploring an analogy between deep networks and dynamical systems, where the growth or decay of local perturbations is characterised by finite-time Lyapunov exponents. We show that the maximal exponent forms geometrical structures in input space, akin to coherent structures in dynamical systems. Ridges of large positive exponents divide input space into different regions that the network associates with different classes. These ridges visualise the geometry that deep networks construct in input space, shedding light on the fundamental mechanisms underlying their learning capabilities.

6 pages, 4 figures

References in corpus (7)

• A Survey of Uncertainty in Deep Neural Networks
• Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
• A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit
• Geometric compression of invariant manifolds in neural nets
• The edge as a Lagrangian Coherent Structure in a high-dimensional state space
• On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity

Cited by in corpus (4)

• Dynamical analysis of a parameter-aware reservoir computer
• Attractor-merging Crises and Intermittency in Reservoir Computing
• Finite-time Lyaponov analysis of a trained reservoir computer
• Network Dynamics-Based Framework for Understanding Deep Neural Networks