5 papers
On the Value of Tokeniser Pretraining in Physics Foundation Models
Hadi Sotoudeh, Payel Mukhopadhyay, Ruben Ohana +4
We investigate the impact of tokeniser pretraining on the accuracy and efficiency of physics emulation. Modern high-resolution simulations produce vast volumes of data spanning div…
Revisiting Deep AC-OPF
Oluwatomisin I. Dada, Neil D. Lawrence
Recent work has proposed machine learning (ML) approaches as fast surrogates for solving AC optimal power flow (AC-OPF), with claims of significant speed-ups and high accuracy. In…
Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvements
Aditya Ravuri, Neil D. Lawrence
We propose a probabilistic interpretation of transformers as unrolled inference steps assuming a probabilistic Laplacian Eigenmaps model from the ProbDR framework. Our derivation s…
Protein Language Model Zero-Shot Fitness Predictions are Improved by Inference-only Dropout
Aditya Ravuri, Neil D. Lawrence
Protein Language Models (PLMs) such as ESM2 have been shown to be capable of zero-shot prediction of critical scalar properties of proteins (fitness). In this work, we show that in…
Towards One Model for Classical Dimensionality Reduction: A Probabilistic Perspective on UMAP and t-SNE
Aditya Ravuri, Neil D. Lawrence
This paper shows that dimensionality reduction methods such as UMAP and t-SNE, can be approximately recast as MAP inference methods corresponding to a model introduced in Ravuri et…