Towards Foundation Models for Materials Science: The Open MatSci ML Toolkit

Abstract

Artificial intelligence and machine learning have shown great promise in their ability to accelerate novel materials discovery. As researchers and domain scientists seek to unify and consolidate chemical knowledge, the case for models with potential to generalize across different tasks within materials science - so-called "foundation models" - grows with ambitions. This manuscript reviews our recent progress with development of Open MatSci ML Toolkit, and details experiments that lay the groundwork for foundation model research and development with our framework. First, we describe and characterize a new pretraining task that uses synthetic data generated from symmetry operations, and reveal complex training dynamics at large scales. Using the pretrained model, we discuss a number of use cases relevant to foundation model development: semantic architecture of datasets, and fine-tuning for property prediction and classification. Our key results show that for simple applications, pretraining appears to provide worse modeling performance than training models from random initialization. However, for more complex instances, such as when a model is required to learn across multiple datasets and types of targets simultaneously, the inductive bias from pretraining provides significantly better performance. This insight will hopefully inform subsequent efforts into creating foundation models for materials science applications.

arXiv comment17 pages, 7 figures, 1 table. Accepted paper/presentation at the AI4Science workshop at Super Computing '23

References in corpus (21)

• Scikit-learn: Machine Learning in Python
• Decoupled Weight Decay Regularization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On the Opportunities and Risks of Foundation Models
• PaLM: Scaling Language Modeling with Pathways
• Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals
• The Open Catalyst 2020 (OC20) Dataset and Community Challenges
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm
• The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts
• Generative adversarial networks (GAN) based efficient sampling of chemical space for inverse design of inorganic materials
• A Data Ecosystem to Support Machine Learning in Materials Science
• GemNet: Universal Directional Graph Neural Networks for Molecules
• Large scale and linear scaling DFT with the CONQUEST code
• Root Mean Square Layer Normalization
• A Survey on Oversmoothing in Graph Neural Networks
• The Materials Simulation Toolkit for Machine Learning (MAST-ML): an automated open source toolkit to accelerate data-driven materials research
• Geometric Algebra Transformer
• The Open MatSci ML Toolkit: A Flexible Framework for Machine Learning in Materials Science
• PhAST: Physics-Aware, Scalable, and Task-specific GNNs for Accelerated Catalyst Design
• A Theory on Adam Instability in Large-Scale Machine Learning

Cited by in corpus (3)

• DPA-2: a large atomic model as a multi-task learner
• Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNN
• Predicting Many Crystal Properties via an Adaptive Transformer-based Framework