Perspective: Energy Landscapes for Machine Learning

Abstract

Machine learning techniques are being increasingly used as flexible non-linear fitting and prediction tools in the physical sciences. Fitting functions that exhibit multiple solutions as local minima can be analysed in terms of the corresponding machine learning landscape. Methods to explore and visualise molecular potential energy landscapes can be applied to these machine learning landscapes to gain new insight into the solution space involved in training and the nature of the corresponding predictions. In particular, we can define quantities analogous to molecular structure, thermodynamics, and kinetics, and relate these emergent properties to the structure of the underlying landscape. This Perspective aims to describe these analogies with examples from recent applications, and suggest avenues for new interdisciplinary research.

41 pages, 25 figures. Accepted for publication in Physical Chemistry Chemical Physics, 2017

References in corpus (7)

• Some Further Results for the Stationary Points and Dynamics of Supercooled Liquids
• Exploring the Free Energy Landscape: From Dynamics to Networks and Back
• Representing molecule-surface interactions with symmetry-adapted neural networks
• Characterizing the network topology of the energy landscapes of atomic clusters
• Inverse Design for Self Assembly via On-the-Fly Optimization
• Energy Landscape of the Finite-Size Mean-field 2-Spin Spherical Model and Topology Trivialization
• On the energy landscape of spherical spin glasses

Cited by in corpus (39)

• Machine learning and the physical sciences
• Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
• Essentially No Barriers in Neural Network Energy Landscape
• Scaling description of generalization with number of parameters in deep learning
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• Characterizing the loss landscape of variational quantum circuits
• Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning
• Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalization
• Image-driven discriminative and generative machine learning algorithms for establishing microstructure-processing relationships
• The Calabi-Yau Landscape: from Geometry, to Physics, to Machine-Learning
• Geometry of energy landscapes and the optimizability of deep neural networks
• Data efficiency and extrapolation trends in neural network interatomic potentials
• The Loss Surface of XOR Artificial Neural Networks
• Artificial Neural Network in Cosmic Landscape
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• Properties of Kinetic Transition Networks for Atomic Clusters and Glassy Solids
• Convergence of Newton-MR under Inexact Hessian Information
• Quantitative Propagation of Chaos for SGD in Wide Neural Networks
• Towards Robust Deep Neural Networks
• A Local Optima Network View of Real Function Fitness Landscapes
• Taxonomizing local versus global structure in neural network loss landscapes
• Is SGD a Bayesian sampler? Well, almost
• Interpreting Deep Learning: The Machine Learning Rorschach Test?
• When you can't count, sample! Computable entropies beyond equilibrium from basin volumes
• Self-assembly of two-dimensional, amorphous materials on a liquid substrate
• Thermodynamics of the Ising model encoded in restricted Boltzmann machines
• Glass--like transition described by toppling of stability hierarchy
• Data-driven effective model shows a liquid-like deep learning
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• Black holes and the loss landscape in machine learning
• Visualizing Free Energy Landscapes for Four Hard Disks
• Emergent universal long-range structure in random-organizing systems
• Dissecting a Small Artificial Neural Network
• Physics-Inspired Interpretability Of Machine Learning Models
• The energy landscape of a simple neural network
• TATi-Thermodynamic Analytics ToolkIt: TensorFlow-based software for posterior sampling in machine learning applications
• Complete Realization of Energy Landscape and Non-equilibrium Trapping Dynamics in Spin Glass and Optimization Problem