Data efficiency and extrapolation trends in neural network interatomic potentials

Abstract

Over the last few years, key architectural advances have been proposed for neural network interatomic potentials (NNIPs), such as incorporating message-passing networks, equivariance, or many-body expansion terms. Although modern NNIP models exhibit small differences in energy/forces errors, improvements in accuracy are still considered the main target when developing new NNIP architectures. In this work, we show how architectural and optimization choices influence the generalization of NNIPs, revealing trends in molecular dynamics (MD) stability, data efficiency, and loss landscapes. Using the 3BPA dataset, we show that test errors in NNIP follow a scaling relation and can be robust to noise, but cannot predict MD stability in the high-accuracy regime. To circumvent this problem, we propose the use of loss landscape visualizations and a metric of loss entropy for predicting the generalization power of NNIPs. With a large-scale study on NequIP and MACE, we show that the loss entropy predicts out-of-distribution error and MD stability despite being computed only on the training set. Using this probe, we demonstrate how the choice of optimizers, loss function weighting, data normalization, and other architectural decisions influence the extrapolation behavior of NNIPs. Finally, we relate loss entropy to data efficiency, demonstrating that flatter landscapes also predict learning curve slopes. Our work provides a deep learning justification for the extrapolation performance of many common NNIPs, and introduces tools beyond accuracy metrics that can be used to inform the development of next-generation models.

References in corpus (12)

• ANI-1: An extensible neural network potential with DFT accuracy at force field computational cost
• On the Convergence of Adam and Beyond
• Machine Learning Unifies the Modelling of Materials and Molecules
• Exploring Generalization in Deep Learning
• Qualitatively characterizing neural network optimization problems
• Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations
• Fantastic Generalization Measures and Where to Find Them
• How to validate machine-learned interatomic potentials
• Exploring the Limits of Large Scale Pre-training
• ForceNet: A Graph Neural Network for Large-Scale Quantum Calculations
• Data efficiency and extrapolation trends in neural network interatomic potentials
• Exploring loss function topology with cyclical learning rates

Cited by in corpus (4)

• TorchMD-Net 2.0: Fast Neural Network Potentials for Molecular Simulations
• Data efficiency and extrapolation trends in neural network interatomic potentials
• Enhanced sampling of robust molecular datasets with uncertainty-based collective variables
• Breaking scaling relations with inverse catalysts: a machine learning exploration of trends in $\mathrm{C}{\mathrm{O}}_2$ hydrogenation energy barriers