2.9k citations · 12k across the 144 of their papers we have counts for
73 papers · 1 filter
Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Loss
Yi Ren, Wenlong Deng, Guanzhe Hong +2
Modern language models are likely to be updated throughout their lifetime rather than trained once and frozen. Each update therefore participates in a recurring cycle: decide which…
The Neural Tangent Kernel for Classification
Jonathan Plenk, Sergio Calvo-Ordonez, Alvaro Cartea +3
In wide neural networks, the Neural Tangent Kernel (NTK) remains approximately constant during training, providing a powerful theoretical tool for studying training dynamics, gener…
Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networks
Yoav Gelberg, Tycho F. A. van der Ouderaa, Mark van der Wilk +1
Weight space symmetries in neural network architectures, such as permutation symmetries in MLPs, give rise to Bayesian neural network (BNN) posteriors with many equivalent modes. T…
The Benefits and Risks of Transductive Approaches for AI Fairness
Muhammed Razzak, Andreas Kirsch, Yarin Gal
Recently, transductive learning methods, which leverage holdout sets during training, have gained popularity for their potential to improve speed, accuracy, and fairness in machine…
Can Active Sampling Reduce Causal Confusion in Offline Reinforcement Learning?
Gunshi Gupta, Tim G. J. Rudner, Rowan Thomas McAllister +2
Causal confusion is a phenomenon where an agent learns a policy that reflects imperfect spurious correlations in the data. Such a policy may falsely appear to be optimal during tra…
Fine-tuning can cripple your foundation model; preserving features may be the solution
Jishnu Mukhoti, Yarin Gal, Philip H. S. Torr +1
Pre-trained foundation models, due to their enormous capacity and exposure to vast amounts of data during pre-training, are known to have learned plenty of real-world concepts. An…
BatchGFN: Generative Flow Networks for Batch Active Learning
Shreshth A. Malik, Salem Lahlou, Andrew Jesson +5
We introduce BatchGFN -- a novel approach for pool-based active learning that uses generative flow networks to sample sets of data points proportional to a batch reward. With an ap…
ReLU to the Rescue: Improve Your On-Policy Actor-Critic with Positive Advantages
Andrew Jesson, Chris Lu, Gunshi Gupta +4
This paper proposes a step toward approximate Bayesian inference in on-policy actor-critic deep reinforcement learning. It is implemented through three changes to the Asynchronous…
The Curse of Recursion: Training on Generated Data Makes Models Forget
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao +3
Stable Diffusion revolutionised image creation from descriptive text. GPT-2, GPT-3(.5) and GPT-4 demonstrated astonishing performance across a variety of language tasks. ChatGPT in…
Prediction-Oriented Bayesian Active Learning
Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar +3
Information-theoretic approaches to active learning have traditionally focused on maximising the information gathered about the model parameters, most commonly by optimising the BA…
Differentiable Multi-Target Causal Bayesian Experimental Design
Yashas Annadani, Panagiotis Tigas, Desi R. Ivanova +4
We introduce a gradient-based approach for the problem of Bayesian optimal experimental design to learn causal models in a batch setting -- a critical component for causal discover…
Exploring Low Rank Training of Deep Neural Networks
Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh +3
Training deep neural networks in low rank, i.e. with factorised layers, is of particular interest to the community: it offers efficiency over unfactorised training in terms of both…
Unifying Approaches in Active Learning and Active Sampling via Fisher Information and Information-Theoretic Quantities
Andreas Kirsch, Yarin Gal
Recently proposed methods in data subset selection, that is active learning and active sampling, use Fisher information, Hessians, similarity matrices based on gradients, and gradi…
Plex: Towards Reliability using Pretrained Large Model Extensions
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry +23
A recent trend in artificial intelligence is the use of pretrained models for language and vision tasks, which have achieved extraordinary performance but also puzzling failures. P…
Learning Dynamics and Generalization in Reinforcement Learning
Clare Lyle, Mark Rowland, Will Dabney +2
Solving a reinforcement learning (RL) problem poses two competing challenges: fitting a potentially discontinuous value function, and generalizing well to new observations. In this…
Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt
Sören Mindermann, Jan Brauner, Muhammed Razzak +8
Training on web-scale data can take months. But most computation and time is wasted on redundant and noisy points that are already learnt or not learnable. To accelerate training,…
Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval
Pascal Notin, Mafalda Dias, Jonathan Frazer +4
The ability to accurately model the fitness landscape of protein sequences is critical to a wide range of applications, from quantifying the effects of human variants on disease li…
Marginal and Joint Cross-Entropies & Predictives for Online Bayesian Inference, Active Learning, and Active Sampling
Andreas Kirsch, Jannik Kossen, Yarin Gal
Principled Bayesian deep learning (BDL) does not live up to its potential when we only focus on marginal predictive distributions (marginal predictives). Recent works have highligh…
Scalable Sensitivity and Uncertainty Analysis for Causal-Effect Estimates of Continuous-Valued Interventions
Andrew Jesson, Alyson Douglas, Peter Manshausen +5
Estimating the effects of continuous-valued interventions from observational data is a critically important task for climate science, healthcare, and economics. Recent work focuses…
Interventions, Where and How? Experimental Design for Causal Models at Scale
Panagiotis Tigas, Yashas Annadani, Andrew Jesson +3
Causal discovery from observational and interventional data is challenging due to limited data and non-identifiability: factors that introduce uncertainty in estimating the underly…
Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients
Milad Alizadeh, Shyam A. Tailor, Luisa M Zintgraf +4
Pruning neural networks at initialization would enable us to find sparse models that retain the accuracy of the original network while consuming fewer computational resources for t…
Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation
Jannik Kossen, Sebastian Farquhar, Yarin Gal +1
We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels ar…
A Note on "Assessing Generalization of SGD via Disagreement"
Andreas Kirsch, Yarin Gal
Several recent works find empirically that the average test error of deep neural networks can be estimated via the prediction disagreement of models, which does not require labels.…
DARTS without a Validation Set: Optimizing the Marginal Likelihood
Miroslav Fil, Binxin Ru, Clare Lyle +1
The success of neural architecture search (NAS) has historically been limited by excessive compute requirements. While modern weight-sharing NAS methods such as DARTS are able to f…