A critical examination of robustness and generalizability of machine learning prediction of materials properties

Abstract

Recent advances in machine learning (ML) methods have led to substantial improvement in materials property prediction against community benchmarks, but an excellent benchmark score may not imply good generalization of performance. Here we show that ML models trained on the Materials Project 2018 (MP18) dataset can have severely degraded prediction performance on new compounds in the Materials Project 2021 (MP21) dataset. We document performance degradation in graph neural networks and traditional descriptor-based ML models for both quantitative and qualitative predictions. We find the source of the predictive degradation is due to the distribution shift between the MP18 and MP21 versions. This is revealed by the uniform manifold approximation and projection (UMAP) of the feature space. We then show that the performance degradation issue can be foreseen using a few simple tools. Firstly, the UMAP can be used to investigate the connectivity and relative proximity of the training and test data within feature space. Secondly, the disagreement between multiple ML models on the test data can illuminate out-of-distribution samples. We demonstrate that the simple yet efficient UMAP-guided and query-by-committee acquisition strategies can greatly improve prediction accuracy through adding only 1~\% of the test data. We believe this work provides valuable insights for building materials databases and ML models that enable better prediction robustness and generalizability.

References in corpus (11)

• A General-Purpose Machine Learning Framework for Predicting Properties of Inorganic Materials
• Recent Advances and Applications of Deep Learning Methods in Materials Science
• A Universal Graph Deep Learning Interatomic Potential for the Periodic Table
• Atomistic Line Graph Neural Network for Improved Materials Property Predictions
• Opportunities and Challenges for Machine Learning in Materials Science
• On scientific understanding with artificial intelligence
• Unified Graph Neural Network Force-field for the Periodic Table
• Density of States Prediction for Materials Discovery via Contrastive Learning from Probabilistic Embeddings
• AtomSets -- A Hierarchical Transfer Learning Framework for Small and Large Materials Datasets
• Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet
• Formula graph self-attention network for representation-domain independent materials discovery

Cited by in corpus (10)

• Performance Assessment of Universal Machine Learning Interatomic Potentials: Challenges and Directions for Materials' Surfaces
• On the redundancy in large material datasets: efficient and robust learning with less data
• Probing out-of-distribution generalization in machine learning for materials
• JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods
• ET-AL: Entropy-Targeted Active Learning for Bias Mitigation in Materials Data
• Efficient first principles based modeling via machine learning: from simple representations to high entropy materials
• Impact of Data Bias on Machine Learning for Crystal Compound Synthesizability Predictions
• Classification-based detection and quantification of cross-domain data bias in materials discovery
• Building Materials Genome from Ground-State Configuration to Engineering Advance
• Electron-Informed Coarse-Graining Molecular Representation Learning for Real-World Molecular Physics