Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm

Abstract

We present a benchmark test suite and an automated machine learning procedure for evaluating supervised machine learning (ML) models for predicting properties of inorganic bulk materials. The test suite, Matbench, is a set of 13 ML tasks that range in size from 312 to 132k samples and contain data from 10 density functional theory-derived and experimental sources. Tasks include predicting optical, thermal, electronic, thermodynamic, tensile, and elastic properties given a materials composition and/or crystal structure. The reference algorithm, Automatminer, is a highly-extensible, fully-automated ML pipeline for predicting materials properties from materials primitives (such as composition and crystal structure) without user intervention or hyperparameter tuning. We test Automatminer on the Matbench test suite and compare its predictive power with state-of-the-art crystal graph neural networks and a traditional descriptor-based Random Forest model. We find Automatminer achieves the best performance on 8 of 13 tasks in the benchmark. We also show our test suite is capable of exposing predictive advantages of each algorithm - namely, that crystal graph methods appear to outperform traditional machine learning methods given ~10^4 or greater data points. The pre-processed, ready-to-use Matbench tasks and the Automatminer source code are open source and available online (http://hackingmaterials.lbl.gov/automatminer/). We encourage evaluating new materials ML algorithms on the MatBench benchmark and comparing them against the latest version of Automatminer.

Main text, supplemental info

References in corpus (4)

• Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties
• A General-Purpose Machine Learning Framework for Predicting Properties of Inorganic Materials
• Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals
• Machine learning with force-field inspired descriptors for materials: fast screening and mapping energy landscape

Cited by in corpus (38)

• Advances of Machine Learning in Materials Science: Ideas and Techniques
• Modelling the dielectric constants of crystals using machine learning
• JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods
• Calibrating DFT formation enthalpy calculations by multi-fidelity machine learning
• Recent progress in the JARVIS infrastructure for next-generation data-driven materials design
• Accelerating Material Design with the Generative Toolkit for Scientific Discovery
• Insights into cation ordering of double perovskite oxides from machine learning and causal relations
• DenseGNN: universal and scalable deeper graph neural networks for high-performance property prediction in crystals and molecules
• A rule-free workflow for the automated generation of databases from scientific literature
• An equivariant graph neural network for the elasticity tensors of all seven crystal systems
• A Quantum-Chemical Bonding Database for Solid-State Materials
• Robust model benchmarking and bias-imbalance in data-driven materials science: a case study on MODNet
• High-dimensional Bayesian Optimization of Hyperparameters for an Attention-based Network to Predict Materials Property: a Case Study on CrabNet using Ax and SAASBO
• Formula graph self-attention network for representation-domain independent materials discovery
• Reproducibility in Computational Materials Science: Lessons from 'A General-Purpose Machine Learning Framework for Predicting Properties of Inorganic Materials'
• Symbolic Regression as Feature Engineering Method for Machine and Deep Learning Regression Tasks
• Optimal pre-train/fine-tune strategies for accurate material property predictions
• LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction
• Predicting ionic conductivity in solids from the machine-learned potential energy landscape
• Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems
• Machine-learning correction to density-functional crystal structure optimization
• Accelerating Material Property Prediction using Generically Complete Isometry Invariants
• Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions
• Towards Foundation Models for Materials Science: The Open MatSci ML Toolkit
• Material Property Prediction using Graphs based on Generically Complete Isometry Invariants
• Self-supervised graph neural networks for accurate prediction of Néel temperature
• Optical materials discovery and design with federated databases and machine learning
• PINK: physical-informed machine learning for lattice thermal conductivity
• Benchmarking Inverse Optimization Algorithms for Materials Design
• Topological descriptors for the electron density of inorganic solids
• Automated Machine Learning Pipeline: Large Language Models-Assisted Automated Dataset Generation for Training Machine-Learned Interatomic Potentials
• Structure-Driven Prediction of Magnetic Order in Uranium Compounds
• Self-supervised Representations and Node Embedding Graph Neural Networks for Accurate and Multi-scale Analysis of Materials
• General Learning of the Electric Response of Inorganic Materials
• Predicting Many Crystal Properties via an Adaptive Transformer-based Framework
• Entropy-based Active Learning of Graph Neural Network Surrogate Models for Materials Properties
• Machine learning-driven elasticity prediction in advanced inorganic materials via convolutional neural networks
• Atoms to Processes: The Role of Artificial Intelligence and Machine Learning in Chemical Engineering