SAIBench: Benchmarking AI for Science
arXiv:2206.05418 · doi:10.1016/j.tbench.2022.100063
Abstract
Scientific research communities are embracing AI-based solutions to target tractable scientific tasks and improve research workflows. However, the development and evaluation of such solutions are scattered across multiple disciplines. We formalize the problem of scientific AI benchmarking, and propose a system called SAIBench in the hope of unifying the efforts and enabling low-friction on-boarding of new disciplines. The system approaches this goal with SAIL, a domain-specific language to decouple research problems, AI models, ranking criteria, and software/hardware configuration into reusable modules. We show that this approach is flexible and can adapt to problems, AI models, and evaluation methods defined in different perspectives. The project homepage is https://www.computercouncil.org/SAIBench
arXiv commentPublished in BenchCouncil Transactions on Benchmarks, Standards and Evaluations (TBench)
References in corpus (6)
- PhysNet: A Neural Network for Predicting Energies, Forces, Dipole Moments and Partial Charges
- Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations
- NWChem: Past, Present, and Future
- Deep Potential: a general representation of a many-body potential energy surface
- Review: Deep Learning in Electron Microscopy
- MLHarness: A Scalable Benchmarking System for MLCommons