Metric Flows with Neural Networks
arXiv:2310.19870 · doi:10.1088/2632-2153/ad8533
Abstract
We develop a general theory of flows in the space of Riemannian metrics induced by neural network gradient descent. This is motivated in part by recent advances in approximating Calabi-Yau metrics with neural networks and is enabled by recent advances in understanding flows in the space of neural networks. We derive the corresponding metric flow equations, which are governed by a metric neural tangent kernel, a complicated, non-local object that evolves in time. However, many architectures admit an infinite-width limit in which the kernel becomes fixed and the dynamics simplify. Additional assumptions can induce locality in the flow, which allows for the realization of Perelman's formulation of Ricci flow that was used to resolve the 3d Poincaré conjecture. We demonstrate that such fixed kernel regimes lead to poor learning of numerical Calabi-Yau metrics, as is expected since the associated neural networks do not learn features. Conversely, we demonstrate that well-learned numerical metrics at finite-width exhibit an evolving metric-NTK, associated with feature learning. Our theory of neural network metric flows therefore explains why neural networks are better at learning Calabi-Yau metrics than fixed kernel methods, such as the Ricci flow.
arXiv comment32 pages + references. Published in Machine Learning: Science and Technology
References in corpus (20)
- Scikit-learn: Machine Learning in Python
- Practical Bayesian Optimization of Machine Learning Algorithms
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
- Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
- Notes on Perelman's papers
- Neural Networks and Quantum Field Theory
- Some numerical results in complex differential geometry
- On Algorithmic Universality in F-theory Compactifications
- Neural Tangents: Fast and Easy Infinite Neural Networks in Python
- Tensor Programs II: Neural Tangent Kernel for Any Architecture
- Finite Versus Infinite Neural Networks: an Empirical Study
- Numerical Calabi-Yau metrics from holomorphic networks
- CYJAX: A package for Calabi-Yau metrics with JAX
- -properness of Mabuchi's -energy
- Fast Finite Width Neural Tangent Kernel
- Level Crossings, Attractor Points and Complex Multiplication
- Neural Network Approximations for Calabi-Yau Metrics