Is novelty predictable?

Abstract

Machine learning-based design has gained traction in the sciences, most notably in the design of small molecules, materials, and proteins, with societal implications spanning drug development and manufacturing, plastic degradation, and carbon sequestration. When designing objects to achieve novel property values with machine learning, one faces a fundamental challenge: how to push past the frontier of current knowledge, distilled from the training data into the model, in a manner that rationally controls the risk of failure. If one trusts learned models too much in extrapolation, one is likely to design rubbish. In contrast, if one does not extrapolate, one cannot find novelty. Herein, we ponder how one might strike a useful balance between these two extremes. We focus in particular on designing proteins with novel property values, although much of our discussion addresses machine learning-based design more broadly.

References in corpus (34)

• Quantification of the effect of mutations using a global probability model of natural sequence variation
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Machine learning-assisted directed protein evolution with combinatorial libraries
• Group Equivariant Convolutional Networks
• Coevolutionary landscape inference and the context-dependence of mutations in beta-lactamase TEM-1
• Batch Bayesian Optimization via Local Penalization
• How pairwise coevolutionary models capture the collective residue variability in proteins
• Are Emergent Abilities of Large Language Models a Mirage?
• Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval
• The Parallel Knowledge Gradient Method for Batch Bayesian Optimization
• e3nn: Euclidean Neural Networks
• Efficient generative modeling of protein sequences using simple autoregressive models
• Parallelizing Exploration-Exploitation Tradeoffs with Gaussian Process Bandit Optimization
• Maximizing acquisition functions for Bayesian optimization
• Parallel Predictive Entropy Search for Batch Global Optimization of Expensive Objective Functions
• RITA: a Study on Scaling Up Generative Protein Sequence Models
• Conformal Prediction Under Covariate Shift
• ProGen2: Exploring the Boundaries of Protein Language Models
• Bayes, Jeffreys, Prior Distributions and the Philosophy of Statistics
• Conditioning by adaptive sampling for robust design
• Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design
• AdaLead: A simple and robust adaptive greedy search algorithm for sequence design
• Distribution-Free Predictive Inference For Regression
• Computational protein design with evolutionary-based and physics-inspired modeling: current and future synergies
• Adaptive Conformal Inference Under Distribution Shift
• Accelerating Bayesian Optimization for Biological Sequence Design with Denoising Autoencoders
• Autofocused oracles for model-based design
• Batched Stochastic Bayesian Optimization via Combinatorial Constraints Design
• Distribution-free uncertainty quantification for classification under label shift
• Conformal prediction beyond exchangeability
• Telescoping Density-Ratio Estimation
• Density Ratio Estimation via Infinitesimal Classification
• Bayesian Optimization with Conformal Prediction Sets
• Augmenting Neural Networks with Priors on Function Values