JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods

Abstract

Lack of rigorous reproducibility and validation are major hurdles for scientific development across many fields. Materials science in particular encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with both perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC) and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data-points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard

References in corpus (58)

• Quantum ESPRESSO: a modular and open-source software project for quantum simulations of materials
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Restoring the density-gradient expansion for exchange in solids and surfaces
• Generalized gradient approximation for solids and their surfaces
• Van der Waals density functionals applied to solids
• A variational eigenvalue solver on a quantum processor
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties
• On representing chemical environments
• Electronic Structure Calculations with Dynamical Mean-Field Theory: A Spectral Density Functional Approach
• Deep Potential Molecular Dynamics: a scalable model with the accuracy of quantum mechanics
• SchNet - a deep learning architecture for molecules and materials
• A General-Purpose Machine Learning Framework for Predicting Properties of Inorganic Materials
• AFLOW: An automatic framework for high-throughput materials discovery
• Machine Learning of Accurate Energy-Conserving Molecular Force Fields
• Recent Advances and Applications of Deep Learning Methods in Materials Science
• A Universal Graph Deep Learning Interatomic Potential for the Periodic Table
• A Performance and Cost Assessment of Machine Learning Interatomic Potentials
• Heterogeneous Parallelization and Acceleration of Molecular Dynamics Simulations in GROMACS
• Precision and efficiency in solid-state pseudopotential calculations
• GuacaMol: Benchmarking Models for De Novo Molecular Design
• The Open Catalyst 2020 (OC20) Dataset and Community Challenges
• Atomistic Line Graph Neural Network for Improved Materials Property Predictions
• Validating quantum computers using randomized model circuits
• The Joint Automated Repository for Various Integrated Simulations (JARVIS) for data-driven materials design
• Variational Quantum Computation of Excited States
• Benchmarking Materials Property Prediction Methods: The Matbench Test Set and Automatminer Reference Algorithm
• The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts
• Kohn-Sham potential with discontinuity for band gap materials
• A critical examination of compound stability predictions from machine-learned formation energies
• sGDML: Constructing Accurate and Data Efficient Molecular Force Fields Using Machine Learning
• PennyLane: Automatic differentiation of hybrid quantum-classical computations
• Convergence and machine learning predictions of Monkhorst-Pack k-points and plane-wave cut-off in high-throughput DFT calculations
• Accurate Force Field for Molybdenum by Machine Learning Large Materials Data
• AtomAI: A Deep Learning Framework for Analysis of Image and Spectroscopy Data in (Scanning) Transmission Electron Microscopy and Beyond
• A critical examination of robustness and generalizability of machine learning prediction of materials properties
• State Predictive Information Bottleneck
• Unified Graph Neural Network Force-field for the Periodic Table
• On the redundancy in large material datasets: efficient and robust learning with less data
• Automated discovery of a robust interatomic potential for aluminum
• How to verify the precision of density-functional-theory implementations via reproducible and universal workflows
• Graph Neural Network Predictions of Metal Organic Framework CO2 Adsorption Properties
• Olympus: a benchmarking framework for noisy optimization and experiment planning
• Tequila: A platform for rapid development of quantum algorithms
• Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models
• ChemNLP: A Natural Language Processing based Library for Materials Chemistry Text Data
• IRNet: A General Purpose Deep Residual Regression Framework for Materials Discovery
• Ferromagnetic half levitation of LK-99-like synthetic samples
• Recent progress in the JARVIS infrastructure for next-generation data-driven materials design
• Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spaces
• A Deep-learning Model for Fast Prediction of Vacancy Formation in Diverse Materials
• Implementing graph neural networks with TensorFlow-Keras
• Reproducibility of density functional approximations: how new functionals should be reported
• The OpenKIM Processing Pipeline: A Cloud-Based Automatic Materials Property Computation Engine
• AtomVision: A machine vision library for atomistic images
• Benchmark tests of atom segmentation deep learning models with a consistent dataset
• Differentiable quantum computational chemistry with PennyLane
• Materials Transformers Language Models for Generative Materials Design: a benchmark study

Cited by in corpus (10)

• Recent progress in the JARVIS infrastructure for next-generation data-driven materials design
• InterMat: Accelerating Band Offset Prediction in Semiconductor Interfaces with DFT and Deep Learning
• Data-driven Design of High Pressure Hydride Superconductors using DFT and Deep Learning
• LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction
• Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems
• The JARVIS Infrastructure is All You Need for Materials Design
• The Enduring Relevance of Semiempirical Quantum Mechanics
• Data Fusion of Deep Learned Molecular Embeddings for Property Prediction
• CHILI: Chemically-Informed Large-scale Inorganic Nanomaterials Dataset for Advancing Graph Machine Learning
• Automated Prediction of Thermodynamic Properties via Bayesian Free-Energy Reconstruction from Molecular Dynamics