Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks

Abstract

Deep learning architectures have proved versatile in a number of drug discovery applications, including the modelling of in vitro compound activity. While controlling for prediction confidence is essential to increase the trust, interpretability and usefulness of virtual screening models in drug discovery, techniques to estimate the reliability of the predictions generated with deep learning networks remain largely underexplored. Here, we present Deep Confidence, a framework to compute valid and efficient confidence intervals for individual predictions using the deep learning technique Snapshot Ensembling and conformal prediction. Specifically, Deep Confidence generates an ensemble of deep neural networks by recording the network parameters throughout the local minima visited during the optimization phase of a single neural network. This approach serves to derive a set of base learners (i.e., snapshots) with comparable predictive power on average, that will however generate slightly different predictions for a given instance. The variability across base learners and the validation residuals are in turn harnessed to compute confidence intervals using the conformal prediction framework. Using a set of 24 diverse IC50 data sets from ChEMBL 23, we show that Snapshot Ensembles perform on par with Random Forest (RF) and ensembles of independently trained deep neural networks. In addition, we find that the confidence regions predicted using the Deep Confidence framework span a narrower set of values. Overall, Deep Confidence represents a highly versatile error prediction framework that can be applied to any deep learning-based application at no extra computational cost.

References in corpus (15)

• Scikit-learn: Machine Learning in Python
• Automatic chemical design using a data-driven continuous representation of molecules
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• On Calibration of Modern Neural Networks
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• A tutorial on conformal prediction
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discovery
• Massively Multitask Networks for Drug Discovery
• Multi-task Neural Networks for QSAR Predictions
• Reliable Prediction Intervals with Regression Neural Networks
• Toxicity Prediction using Deep Learning
• Confidence estimation in Deep Neural networks via density modelling
• Exploring loss function topology with cyclical learning rates

Cited by in corpus (12)

• Opportunities and Challenges for Machine Learning in Materials Science
• Autonomous discovery in the chemical sciences part II: Outlook
• Reliable Prediction Errors for Deep Neural Networks Using Test-Time Dropout
• KekuleScope: prediction of cancer cell line sensitivity and compound potency using convolutional neural networks trained on compound images
• Constrained Bayesian Optimization for Automatic Chemical Design
• Twin Neural Network Regression
• Certified machine learning: A posteriori error estimation for physics-informed neural networks
• Ab initio machine learning in chemical compound space
• Efficient and minimal length parametric conformal prediction regions
• Training conformal predictors
• Neural Bootstrapper
• Leveraging Uncertainty from Deep Learning for Trustworthy Materials Discovery Workflows