Data efficiency and extrapolation trends in neural network interatomic potentials

Abstract

Over the last few years, key architectural advances have been proposed for neural network interatomic potentials (NNIPs), such as incorporating message-passing networks, equivariance, or many-body expansion terms. Although modern NNIP models exhibit small differences in energy/forces errors, improvements in accuracy are still considered the main target when developing new NNIP architectures. In this work, we show how architectural and optimization choices influence the generalization of NNIPs, revealing trends in molecular dynamics (MD) stability, data efficiency, and loss landscapes. Using the 3BPA dataset, we show that test errors in NNIP follow a scaling relation and can be robust to noise, but cannot predict MD stability in the high-accuracy regime. To circumvent this problem, we propose the use of loss landscape visualizations and a metric of loss entropy for predicting the generalization power of NNIPs. With a large-scale study on NequIP and MACE, we show that the loss entropy predicts out-of-distribution error and MD stability despite being computed only on the training set. Using this probe, we demonstrate how the choice of optimizers, loss function weighting, data normalization, and other architectural decisions influence the extrapolation behavior of NNIPs. Finally, we relate loss entropy to data efficiency, demonstrating that flatter landscapes also predict learning curve slopes. Our work provides a deep learning justification for the extrapolation performance of many common NNIPs, and introduces tools beyond accuracy metrics that can be used to inform the development of next-generation models.

References in corpus (44)

• Neural Message Passing for Quantum Chemistry
• Gaussian Approximation Potentials: the accuracy of quantum mechanics, without the electrons
• Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics
• SchNet - a deep learning architecture for molecules and materials
• ANI-1: An extensible neural network potential with DFT accuracy at force field computational cost
• E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials
• On the Convergence of Adam and Beyond
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• Universally Sloppy Parameter Sensitivities in Systems Biology
• Machine Learning of Accurate Energy-Conserving Molecular Force Fields
• A Universal Graph Deep Learning Interatomic Potential for the Periodic Table
• A Performance and Cost Assessment of Machine Learning Interatomic Potentials
• Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems
• Machine Learning Unifies the Modelling of Materials and Molecules
• Visualizing the Loss Landscape of Neural Nets
• Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
• Ab initio thermodynamics of liquid and solid water
• Directional Message Passing for Molecular Graphs
• FCHL revisited: faster and more accurate quantum machine learning
• Hierarchical modeling of molecular energies using a deep neural network
• MACE: Higher Order Equivariant Message Passing Neural Networks for Fast and Accurate Force Fields
• Exploring Generalization in Deep Learning
• Equivariant message passing for the prediction of tensorial properties and molecular spectra
• Extending the Accuracy of the SNAP Interatomic Potential Form
• Qualitatively characterizing neural network optimization problems
• Essentially No Barriers in Neural Network Energy Landscape
• Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations
• The loss surface of deep and wide neural networks
• Fantastic Generalization Measures and Where to Find Them
• Cormorant: Covariant Molecular Neural Networks
• How to validate machine-learned interatomic potentials
• GemNet: Universal Directional Graph Neural Networks for Molecules
• Perspective: Energy Landscapes for Machine Learning
• The Design Space of E(3)-Equivariant Atom-Centered Interatomic Potentials
• Automated discovery of a robust interatomic potential for aluminum
• Shaping the learning landscape in neural networks around wide flat minima
• Differentiable sampling of molecular geometries with uncertainty-based adversarial attacks
• Roaming leads to amino acid photodamage: A deep learning study of tyrosine
• An empirical analysis of the optimization of deep network loss surfaces
• Exploring the Limits of Large Scale Pre-training
• ForceNet: A Graph Neural Network for Large-Scale Quantum Calculations
• Bayesian, frequentist, and information geometric approaches to parametric uncertainty quantification of classical empirical interatomic potentials
• Data efficiency and extrapolation trends in neural network interatomic potentials
• Exploring loss function topology with cyclical learning rates

Cited by in corpus (4)

• TorchMD-Net 2.0: Fast Neural Network Potentials for Molecular Simulations
• Data efficiency and extrapolation trends in neural network interatomic potentials
• Enhanced sampling of robust molecular datasets with uncertainty-based collective variables
• Breaking scaling relations with inverse catalysts: a machine learning exploration of trends in $\mathrm{C}{\mathrm{O}}_2$ hydrogenation energy barriers