A community-powered search of machine learning strategy space to find NMR property prediction models

Abstract

The rise of machine learning (ML) has created an explosion in the potential strategies for using data to make scientific predictions. For physical scientists wishing to apply ML strategies to a particular domain, it can be difficult to assess in advance what strategy to adopt within a vast space of possibilities. Here we outline the results of an online community-powered effort to swarm search the space of ML strategies and develop algorithms for predicting atomic-pairwise nuclear magnetic resonance (NMR) properties in molecules. Using an open-source dataset, we worked with Kaggle to design and host a 3-month competition which received 47,800 ML model predictions from 2,700 teams in 84 countries. Within 3 weeks, the Kaggle community produced models with comparable accuracy to our best previously published "in-house" efforts. A meta-ensemble model constructed as a linear combination of the top predictions has a prediction accuracy which exceeds that of any individual model, 7-19x better than our previous state-of-the-art. The results highlight the potential of transformer architectures for predicting quantum mechanical (QM) molecular properties.

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Neural Machine Translation by Jointly Learning to Align and Translate
• Decoupled Weight Decay Regularization
• Neural Message Passing for Quantum Chemistry
• Fast and Accurate Modeling of Molecular Atomization Energies with Machine Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• Big Data meets Quantum Chemistry Approximations: The $\Delta$-Machine Learning Approach
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• FCHL revisited: faster and more accurate quantum machine learning
• Alchemical and structural distribution based representation for improved QML
• Chemical Shifts in Molecular Solids by Machine Learning
• Residual Gated Graph ConvNets
• Averaging Weights Leads to Wider Optima and Better Generalization
• Paradigm shift in electron-based crystallography via machine learning
• Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
• Cormorant: Covariant Molecular Neural Networks
• Remote optimization of an ultra-cold atoms experiment by experts and citizen scientists
• NOMAD 2018 Kaggle Competition: Solving Materials Science Challenges Through Crowd Sourcing

Cited by in corpus (5)

• The Open Catalyst 2020 (OC20) Dataset and Community Challenges
• Roadmap on machine learning glassy dynamics
• Ab initio machine learning in chemical compound space
• Revving up 13C NMR shielding predictions across chemical space: Benchmarks for atoms-in-molecules kernel machine learning with new data for 134 kilo molecules
• Orbital Mixer: Using Atomic Orbital Features for Basis Dependent Prediction of Molecular Wavefunctions