Olympus: a benchmarking framework for noisy optimization and experiment planning

Abstract

Research challenges encountered across science, engineering, and economics can frequently be formulated as optimization tasks. In chemistry and materials science, recent growth in laboratory digitization and automation has sparked interest in optimization-guided autonomous discovery and closed-loop experimentation. Experiment planning strategies based on off-the-shelf optimization algorithms can be employed in fully autonomous research platforms to achieve desired experimentation goals with the minimum number of trials. However, the experiment planning strategy that is most suitable to a scientific discovery task is a priori unknown while rigorous comparisons of different strategies are highly time and resource demanding. As optimization algorithms are typically benchmarked on low-dimensional synthetic functions, it is unclear how their performance would translate to noisy, higher-dimensional experimental tasks encountered in chemistry and materials science. We introduce Olympus, a software package that provides a consistent and easy-to-use framework for benchmarking optimization algorithms against realistic experiments emulated via probabilistic deep-learning models. Olympus includes a collection of experimentally derived benchmark sets from chemistry and materials science and a suite of experiment planning strategies that can be easily accessed via a user-friendly python interface. Furthermore, Olympus facilitates the integration, testing, and sharing of custom algorithms and user-defined datasets. In brief, Olympus mitigates the barriers associated with benchmarking optimization algorithms on realistic experimental scenarios, promoting data sharing and the creation of a standard framework for evaluating the performance of experiment planning strategies

arXiv comment15 pages, 4 figures, 4 tables (with SI: 22 pages, 11 figures, 15 tables). Changes: minor fixes to text and references. Two paragraphs added in Sec. III

References in corpus (9)

• Global Optimization by Basin-Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Containing up to 110 Atoms
• Fast and Accurate Modeling of Molecular Atomization Energies with Machine Learning
• GuacaMol: Benchmarking Models for De Novo Molecular Design
• Autonomous discovery in the chemical sciences part I: Progress
• Autonomous discovery in the chemical sciences part II: Outlook
• Electronic Spectra from TDDFT and Machine Learning in Chemical Space
• Fast machine-learning online optimization of ultra-cold-atom experiments
• Olympus: a benchmarking framework for noisy optimization and experiment planning
• Nonsymmetric multigrid preconditioning for conjugate gradient methods

Cited by in corpus (7)

• Olympus: a benchmarking framework for noisy optimization and experiment planning
• JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods
• Golem: An algorithm for robust experiment and process optimization
• Graph Machine Learning for Design of High-Octane Fuels
• Designing Materials Acceleration Platforms for Heterogeneous CO2 Photo(thermal)catalysis
• Illustrated tutorial on global optimization in nanophotonics
• Achieving Robustness to Aleatoric Uncertainty with Heteroscedastic Bayesian Optimisation