Toward Accurate Interpretable Predictions of Materials Properties within Transformer Language Models

Abstract

Property prediction accuracy has long been a key parameter of machine learning in materials informatics. Accordingly, advanced models showing state-of-the-art performance turn into highly parameterized black boxes missing interpretability. Here, we present an elegant way to make their reasoning transparent. Human-readable text-based descriptions automatically generated within a suite of open-source tools are proposed as materials representation. Transformer language models pretrained on 2 million peer-reviewed articles take as input well-known terms, e.g., chemical composition, crystal symmetry, and site geometry. Our approach outperforms crystal graph networks by classifying four out of five analyzed properties if one considers all available reference data. Moreover, fine-tuned text-based models show high accuracy in the ultra-small data limit. Explanations of their internal machinery are produced using local interpretability techniques and are faithful and consistent with domain expert rationales. This language-centric framework makes accurate property predictions accessible to people without artificial-intelligence expertise.

arXiv comment17 pages, 5 figures, 1 table

References in corpus (43)

• Scikit-learn: Machine Learning in Python
• Decoupled Weight Decay Regularization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Towards A Rigorous Science of Interpretable Machine Learning
• Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties
• Methods for Interpreting and Understanding Deep Neural Networks
• New Tolerance Factor to Predict the Stability of Perovskite Oxides and Halides
• Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals
• AFLOW: An automatic framework for high-throughput materials discovery
• A Universal Graph Deep Learning Interatomic Potential for the Periodic Table
• Big Data of Materials Science - Critical Role of the Descriptor
• DScribe: Library of Descriptors for Machine Learning in Materials Science
• Atomistic Line Graph Neural Network for Improved Materials Property Predictions
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• GNNExplainer: Generating Explanations for Graph Neural Networks
• Captum: A unified and generic model interpretability library for PyTorch
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• The Joint Automated Repository for Various Integrated Simulations (JARVIS) for data-driven materials design
• Physics-inspired structural representations for molecules and materials
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Understanding Neural Networks through Representation Erasure
• Predicting materials properties without crystal structure: Deep representation learning from stoichiometry
• Interpretable and Explainable Machine Learning for Materials Science and Chemistry
• Machine Learning Energies of 2 M Elpasolite (ABC${}_2$D${}_6$) Crystals
• Accelerating crystal structure prediction by machine-learning interatomic potentials with active learning
• On scientific understanding with artificial intelligence
• Representation of compounds for machine-learning prediction of physical properties
• Symbolic Regression in Materials Science
• Physical descriptor for the Gibbs energy of inorganic crystalline solids and temperature-dependent materials chemistry
• A critical examination of compound stability predictions from machine-learned formation energies
• Beyond scaling relations for the description of catalytic materials
• Parameterized Explainer for Graph Neural Network
• An invertible crystallographic representation for general inverse design of inorganic crystals with targeted properties
• Machine learning with force-field inspired descriptors for materials: fast screening and mapping energy landscape
• Accelerated Discovery of Efficient Solar-cell Materials using Quantum and Machine-learning Methods
• Representations of molecules and materials for interpolation of quantum-mechanical simulations via machine learning
• Computational screening of high-performance optoelectronic materials using OptB88vdW and TB-mBJ formalisms
• Graph convolutional neural networks as "general-purpose" property predictors: the universality and limits of applicability
• BIGDML: Towards Exact Machine Learning Force Fields for Materials
• Relationships between distortions of inorganic framework and band gap of layered hybrid halide perovskites
• Transferable and extensible machine learning derived atomic charges for modeling hybrid nanoporous materials
• Database, Features, and Machine Learning Model to Identify Thermally Driven Metal-Insulator Transition Compounds
• ferret: a Framework for Benchmarking Explainers on Transformers

Cited by in corpus (4)

• Machine Learning - Driven Materials Discovery: Unlocking Next-Generation Functional Materials - A review
• Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie
• LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction
• Coarse-grained crystal graph neural networks for reticular materials design