Most Ligand-Based Classification Benchmarks Reward Memorization Rather than Generalization

Abstract

Undetected overfitting can occur when there are significant redundancies between training and validation data. We describe AVE, a new measure of training-validation redundancy for ligand-based classification problems that accounts for the similarity amongst inactive molecules as well as active. We investigated seven widely-used benchmarks for virtual screening and classification, and show that the amount of AVE bias strongly correlates with the performance of ligand-based predictive methods irrespective of the predicted property, chemical fingerprint, similarity measure, or previously-applied unbiasing techniques. Therefore, it may be that the previously-reported performance of most ligand-based methods can be explained by overfitting to benchmarks rather than good prospective accuracy.

References in corpus (1)

• Molecular Graph Convolutions: Moving Beyond Fingerprints

Cited by in corpus (7)

• Bayesian semi-supervised learning for uncertainty-calibrated prediction of molecular properties and active learning
• Combining docking pose rank and structure with deep learning improves protein-ligand binding mode prediction
• Validating the Validation: Reanalyzing a large-scale comparison of Deep Learning and Machine Learning models for bioactivity prediction
• Quantum Machine Learning Framework for Virtual Screening in Drug Discovery: a Prospective Quantum Advantage
• Random projections and Kernelised Leave One Cluster Out Cross-Validation: Universal baselines and evaluation tools for supervised machine learning for materials properties
• PharmacoNet: Accelerating Large-Scale Virtual Screening by Deep Pharmacophore Modeling
• HD-Bind: Encoding of Molecular Structure with Low Precision, Hyperdimensional Binary Representations