JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods

Abstract

Lack of rigorous reproducibility and validation are major hurdles for scientific development across many fields. Materials science in particular encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with both perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC) and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data-points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard

References in corpus (20)

• Quantum ESPRESSO: a modular and open-source software project for quantum simulations of materials
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Restoring the density-gradient expansion for exchange in solids and surfaces
• Generalized gradient approximation for solids and their surfaces
• A General-Purpose Machine Learning Framework for Predicting Properties of Inorganic Materials
• Accurate Force Field for Molybdenum by Machine Learning Large Materials Data
• A critical examination of robustness and generalizability of machine learning prediction of materials properties
• State Predictive Information Bottleneck
• Unified Graph Neural Network Force-field for the Periodic Table
• On the redundancy in large material datasets: efficient and robust learning with less data
• How to verify the precision of density-functional-theory implementations via reproducible and universal workflows
• Tequila: A platform for rapid development of quantum algorithms
• Alchemy: A Quantum Chemistry Dataset for Benchmarking AI Models
• ChemNLP: A Natural Language Processing based Library for Materials Chemistry Text Data
• IRNet: A General Purpose Deep Residual Regression Framework for Materials Discovery
• Ferromagnetic half levitation of LK-99-like synthetic samples
• Recent progress in the JARVIS infrastructure for next-generation data-driven materials design
• Reproducibility of density functional approximations: how new functionals should be reported
• AtomVision: A machine vision library for atomistic images
• Materials Transformers Language Models for Generative Materials Design: a benchmark study

Cited by in corpus (10)

• Recent progress in the JARVIS infrastructure for next-generation data-driven materials design
• InterMat: Accelerating Band Offset Prediction in Semiconductor Interfaces with DFT and Deep Learning
• Data-driven Design of High Pressure Hydride Superconductors using DFT and Deep Learning
• LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction
• Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems
• The JARVIS Infrastructure is All You Need for Materials Design
• The Enduring Relevance of Semiempirical Quantum Mechanics
• Data Fusion of Deep Learned Molecular Embeddings for Property Prediction
• CHILI: Chemically-Informed Large-scale Inorganic Nanomaterials Dataset for Advancing Graph Machine Learning
• Automated Prediction of Thermodynamic Properties via Bayesian Free-Energy Reconstruction from Molecular Dynamics