Perspective: Energy Landscapes for Machine Learning
arXiv:1703.07915 · doi:10.1039/C7CP01108C
Abstract
Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular potential energy landscapes can be applied to these machine learning landscapes to gain new insight into the solution space involved in training and the nature of the corresponding predictions. In particular, we can define quantities analogous to molecular structure, thermodynamics, and kinetics, and relate these emergent properties to the structure of the underlying landscape. This Perspective aims to describe these analogies with examples from recent applications, and suggest avenues for new interdisciplinary research.
arXiv comment41 pages, 25 figures. Accepted for publication in Physical Chemistry Chemical Physics, 2017
References in corpus (35)
- Emergence of scaling in random networks
- Global Optimization by Basin-Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Containing up to 110 Atoms
- Statistically optimal analysis of samples from multiple equilibrium states
- Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- String Method for the Study of Rare Events
- A Doubly Nudged Elastic Band Method for Finding Transition States
- The double-funnel energy landscape of the 38-atom Lennard-Jones cluster
- Thermodynamics as a theory of decision-making with information processing costs
- Phase changes in 38 atom Lennard-Jones clusters. I: A parallel tempering study in the canonical ensemble
- The network topology of a potential energy landscape: A static scale-free network
- Phase changes in 38 atom Lennard-Jones clusters. II: A parallel tempering study of equilibrium and dynamic properties in the molecular dynamics and microcanonical
- Action-derived molecular dynamics in the study of rare events
- Some Further Results for the Stationary Points and Dynamics of Supercooled Liquids
- Exploring the Free Energy Landscape: From Dynamics to Networks and Back
- Representing molecule-surface interactions with symmetry-adapted neural networks
- The statistics of critical points of Gaussian fields on large-dimensional spaces
- Complexity of random smooth functions on the high-dimensional sphere
- Numerical calculation of granular entropy
- Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
- Characterizing the network topology of the energy landscapes of atomic clusters
- Finding All the Stationary Points of a Potential Energy Landscape via Numerical Polynomial Homotopy Continuation Method
- A novel approach to numerical measurements of the configurational entropy in supercooled liquids
- Direct determination of the size of basins of attraction of jammed solids
- Turning intractable counting into sampling: computing the configurational entropy of three-dimensional jammed packings
- Inverse Design for Self Assembly via On-the-Fly Optimization
- Topology trivialization and large deviations for the minimum in the simplest random optimization
- Visualizing Basins of Attraction for Different Minimization Algorithms
- Simulating structural transitions by direct transition current sampling: the example of LJ38
- The effect of compression on the global optimization of atomic clusters
- Energy Landscape of the Finite-Size Mean-field 3-Spin Spherical Model
- Kinetic Transition Networks for the Thomson Problem and Smale's 7th Problem
- Analysis of cooperativity and localization for atomic rearrangements
- Structural analysis of high-dimensional basins of attraction
- Energy Landscape of the Finite-Size Mean-field 2-Spin Spherical Model and Topology Trivialization
- On the energy landscape of spherical spin glasses
Cited by in corpus (39)
- Machine learning and the physical sciences
- Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
- Essentially No Barriers in Neural Network Energy Landscape
- Scaling description of generalization with number of parameters in deep learning
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- The jamming transition as a paradigm to understand the loss landscape of deep neural networks
- A jamming transition from under- to over-parametrization affects loss landscape and generalization
- Characterizing the loss landscape of variational quantum circuits
- Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
- Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
- Image-driven discriminative and generative machine learning algorithms for establishing microstructure-processing relationships
- The Calabi-Yau Landscape: from Geometry, to Physics, to Machine-Learning
- Geometry of energy landscapes and the optimizability of deep neural networks
- Data efficiency and extrapolation trends in neural network interatomic potentials
- The Loss Surface of XOR Artificial Neural Networks
- Artificial Neural Network in Cosmic Landscape
- Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
- Properties of Kinetic Transition Networks for Atomic Clusters and Glassy Solids
- Convergence of Newton-MR under Inexact Hessian Information
- Quantitative Propagation of Chaos for SGD in Wide Neural Networks
- Towards Robust Deep Neural Networks
- A Local Optima Network View of Real Function Fitness Landscapes
- Taxonomizing local versus global structure in neural network loss landscapes
- Is SGD a Bayesian sampler? Well, almost
- Interpreting Deep Learning: The Machine Learning Rorschach Test?
- When you can't count, sample! Computable entropies beyond equilibrium from basin volumes
- Self-assembly of two-dimensional, amorphous materials on a liquid substrate
- Thermodynamics of the Ising model encoded in restricted Boltzmann machines
- Glass--like transition described by toppling of stability hierarchy
- Data-driven effective model shows a liquid-like deep learning
- Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
- Black holes and the loss landscape in machine learning
- Visualizing Free Energy Landscapes for Four Hard Disks
- Emergent universal long-range structure in random-organizing systems
- Dissecting a Small Artificial Neural Network
- Physics-Inspired Interpretability Of Machine Learning Models
- The energy landscape of a simple neural network
- TATi-Thermodynamic Analytics ToolkIt: TensorFlow-based software for posterior sampling in machine learning applications
- Complete Realization of Energy Landscape and Non-equilibrium Trapping Dynamics in Spin Glass and Optimization Problem