Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet

Abstract

As the number of novel data-driven approaches to material science continues to grow, it is crucial to perform consistent quality, reliability and applicability assessments of model performance. In this paper, we benchmark the Materials Optimal Descriptor Network (MODNet) method and architecture against the recently released MatBench v0.1, a curated test suite of materials datasets. MODNet is shown to outperform current leaders on 6 of the 13 tasks, whilst closely matching the current leaders on a further 2 tasks; MODNet performs particularly well when the number of samples is below 10,000. Attention is paid to two topics of concern when benchmarking models. First, we encourage the reporting of a more diverse set of metrics as it leads to a more comprehensive and holistic comparison of model performance. Second, an equally important task is the uncertainty assessment of a model towards a target domain. Significant variations in validation errors can be observed, depending on the imbalance and bias in the training set (i.e., similarity between training and application space). By using an ensemble MODNet model, confidence intervals can be built and the uncertainty on individual predictions can be quantified. Imbalance and bias issues are often overlooked, and yet are important for successful real-world applications of machine learning in materials science and condensed matter.

References in corpus (13)

• Estimating Mutual Information
• Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• A General-Purpose Machine Learning Framework for Predicting Properties of Inorganic Materials
• Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals
• Data-driven materials science: status, challenges and perspectives
• Machine learning modeling of superconducting critical temperature
• Predicting materials properties without crystal structure: Deep representation learning from stoichiometry
• Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm
• Learning Properties of Ordered and Disordered Materials from Multi-fidelity Data
• MODNet -- accurate and interpretable property predictions for limited materials datasets by feature selection and joint-learning
• Searching for Materials with High Refractive Index and Wide Band Gap: A First-Principles High-Throughput Study
• Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry

Cited by in corpus (11)

• A critical examination of robustness and generalizability of machine learning prediction of materials properties
• Developments and applications of the OPTIMADE API for materials discovery, design, and data exchange
• DenseGNN: universal and scalable deeper graph neural networks for high-performance property prediction in crystals and molecules
• A Quantum-Chemical Bonding Database for Solid-State Materials
• Formula graph self-attention network for representation-domain independent materials discovery
• Impact of Data Bias on Machine Learning for Crystal Compound Synthesizability Predictions
• Predicting Solid State Material Platforms for Quantum Technologies
• Classification-based detection and quantification of cross-domain data bias in materials discovery
• Optical materials discovery and design with federated databases and machine learning
• Accelerating the discovery of high-performance nonlinear optical materials using active learning and high-throughput screening
• General Learning of the Electric Response of Inorganic Materials