Quantifying sources of uncertainty in drug discovery predictions with probabilistic models

Abstract

Knowing the uncertainty in a prediction is critical when making expensive investment decisions and when patient safety is paramount, but machine learning (ML) models in drug discovery typically provide only a single best estimate and ignore all sources of uncertainty. Predictions from these models may therefore be over-confident, which can put patients at risk and waste resources when compounds that are destined to fail are further developed. Probabilistic predictive models (PPMs) can incorporate uncertainty in both the data and model, and return a distribution of predicted values that represents the uncertainty in the prediction. PPMs not only let users know when predictions are uncertain, but the intuitive output from these models makes communicating risk easier and decision making better. Many popular machine learning methods have a PPM or Bayesian analogue, making PPMs easy to fit into current workflows. We use toxicity prediction as a running example, but the same principles apply for all prediction models used in drug discovery. The consequences of ignoring uncertainty and how PPMs account for uncertainty are also described. We aim to make the discussion accessible to a broad non-mathematical audience. Equations are provided to make ideas concrete for mathematical readers (but can be skipped without loss of understanding) and code is available for computational researchers (https://github.com/stanlazic/ML_uncertainty_quantification).

arXiv comment34 pages, 9 figures

References in corpus (13)

• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• Language Models are Few-Shot Learners
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• BART: Bayesian additive regression trees
• Weight Uncertainty in Neural Networks
• Sparsity information and regularization in the horseshoe and other shrinkage priors
• Pareto Smoothed Importance Sampling
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning
• Coresets for Scalable Bayesian Logistic Regression
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Bayesian monotonic errors-in-variables models with applications to pathogen susceptibility testing