High-dimensional Bayesian Optimization of Hyperparameters for an Attention-based Network to Predict Materials Property: a Case Study on CrabNet using Ax and SAASBO
arXiv:2203.12597 · doi:10.1016/j.commatsci.2022.111505
Abstract
Expensive-to-train deep learning models can benefit from an optimization of the hyperparameters that determine the model architecture. We optimize 23 hyperparameters of a materials informatics model, Compositionally-Restricted Attention-Based Network (CrabNet), over 100 adaptive design iterations using two models within the Adaptive Experimentation (Ax) Platform. This includes a recently developed Bayesian optimization (BO) algorithm, sparse axis-aligned subspaces Bayesian optimization (SAASBO), which has shown exciting performance on high-dimensional optimization tasks. Using SAASBO to optimize CrabNet hyperparameters, we demonstrate a new state-of-the-art on the experimental band gap regression task within the materials informatics benchmarking platform, Matbench (~4.5% decrease in mean absolute error (MAE) relative to incumbent). Characteristics of the adaptive design scheme as well as feature importances are described for each of the Ax models. SAASBO has great potential to both improve existing surrogate models, as shown in this work, and in future work, to efficiently discover new, high-performing materials in high-dimensional materials science search spaces.
main: 18 pages, 8 figures. supp: 24 pages, 22 figures. To be submitted to Computational Materials Science
References in corpus (4)
- Fast and Uncertainty-Aware Directional Message Passing for Non-Equilibrium Molecules
- High-Dimensional Bayesian Optimization with Sparse Axis-Aligned Subspaces
- Inverse Design of Composite Metal Oxide Optical Materials based on Deep Transfer Learning
- Uncertainty Toolbox: an Open-Source Library for Assessing, Visualizing, and Improving Uncertainty Quantification