Deep Learning: Hydrodynamics, and Lie-Poisson Hamilton-Jacobi Theory

Abstract

The interpretation of deep learning as a dynamical system has gained a considerable attention in recent years as it provides a promising framework. It allows for the use of existing ideas from established fields of mathematics for studying deep neural networks. In this article we present deep learning as an equivalent hydrodynamics formulation which in fact possess a Lie-Poisson structure and this further allows for using Poisson geometry to study deep learning. This is possible by considering the training of a deep neural network as a stochastic optimal control problem, which is solved using mean-field type control. The optimality conditions for the stochastic optimal control problem yield a system of partial differential equations, which we reduce to a system of equations of quantum hydrodynamics. We further take the hydrodynamics equivalence to show that, with conditions imposed on the network, that it is equivalent to the nonlinear Schrödinger equation. As the such systems have a particular geometric structure, we also present a structure-preserving numerical scheme based on the Hamilton-Jacobi theory and its Lie-Poisson version known as Lie-Poisson Hamilton-Jacobi theory. The choice of this scheme was decided by the fact it can be applied to both deterministic and mean-field type control Hamiltonians. Not only this method eliminates the possibility of the emergence of fictitious solutions, it also eliminates the need for additional constraints when constructing high order methods.

References in corpus (14)

• Sequence to Sequence Learning with Neural Networks
• An integrable shallow water equation with peaked solitons
• Variational Inference with Normalizing Flows
• Multisymplectic geometry, variational integrators, and nonlinear PDEs
• A 2-Component Generalization of the Camassa-Holm Equation and Its Solutions
• The Camassa-Holm equation as a geodesic flow on the diffeomorphism group
• Explicit Lie-Poisson integration and the Euler equations
• Poisson-Lie group of pseudodifferential symbols
• Discrete Hamilton-Jacobi Theory
• Geometric Hydrodynamics via Madelung Transform
• Geometric hydrodynamics and infinite-dimensional Newton's equations
• The Stochastic Energy-Casimir Method
• Euler's fluid equations: Optimal Control vs Optimization
• Integrable Hamiltonian systems on symmetric spaces: Jacobi, Kepler and Moser