Optimal pre-train/fine-tune strategies for accurate material property predictions

Abstract

Overcoming the challenge of limited data availability within materials science is crucial for the broad-based applicability of machine learning within materials science. One pathway to overcome this limited data availability is to use the framework of transfer learning (TL), where a pre-trained (PT) machine learning model (on a larger dataset) can be fine-tuned (FT) on a target (typically smaller) dataset. Our study systematically explores the effectiveness of various PT/FT strategies to learn and predict material properties with limited data. Specifically, we leverage graph neural networks (GNNs) to PT/FT on seven diverse curated materials datasets, encompassing sizes ranging from 941 to 132,752 datapoints. We consider datasets that cover a spectrum of material properties, ranging from band gaps (electronic) to formation energies (thermodynamic) and shear moduli (mechanical). We study the influence of PT and FT dataset sizes, strategies that can be employed for FT, and other hyperparameters on pair-wise TL among the datasets considered. We find our pair-wise PT-FT models to consistently outperform models trained from scratch on the target datasets. Importantly, we develop a GNN framework that is simultaneously PT on multiple properties (MPT), enabling the construction of generalized GNN models. Our MPT models outperform pair-wise PT-FT models on several datasets considered, and more significantly, on a 2D material band gap dataset that is completely out-of-distribution from the PT datasets. Finally, we expect our PT/FT and MPT frameworks to be generalizable to other GNNs and materials properties, which can accelerate materials design and discovery for various applications.

References in corpus (15)

• Overcoming catastrophic forgetting in neural networks
• Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties
• E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials
• Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals
• Atomistic Line Graph Neural Network for Improved Materials Property Predictions
• The Joint Automated Repository for Various Integrated Simulations (JARVIS) for data-driven materials design
• SchNet: A continuous-filter convolutional neural network for modeling quantum interactions
• Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm
• Developing an improved Crystal Graph Convolutional Neural Network framework for accelerated materials discovery
• Glitch Classification and Clustering for LIGO with Deep Transfer Learning
• Deep Transfer Learning: A new deep learning glitch classification method for advanced LIGO
• Deep Transfer Learning for Signal Detection in Ambient Backscatter Communications
• Transfer learning for materials informatics using crystal graph convolutional neural network
• Machine-learning-accelerated simulations to enable automatic surface reconstruction
• A machine learning route between band mapping and band structure

Cited by in corpus (2)

• A literature-derived dataset of migration barriers for quantifying ionic transport in battery materials
• Leveraging transfer learning for accurate estimation of ionic migration barriers in solids