5 papers
Inverse-Free Sparse Variational Gaussian Processes
Stefano Cortinovis, Laurence Aitchison, Stefanos Eleftheriadis +1
Gaussian processes (GPs) offer appealing properties but are costly to train at scale. Sparse variational GP (SVGP) approximations reduce cost yet still rely on Cholesky decompositi…
How to set AdamW's weight decay as you scale model and dataset size
Xi Wang, Laurence Aitchison
The scaling of the optimal AdamW weight decay hyperparameter with model and dataset size is critical as we seek to build larger models, but is poorly understood. We show that weigh…
Residual Stream Analysis with Multi-Layer SAEs
Tim Lawson, Lucy Farnik, Conor Houghton +1
Sparse autoencoders (SAEs) are a promising approach to interpreting the internal representations of transformer language models. However, SAEs are usually trained separately on eac…
Human-inspired Perspectives: A Survey on AI Long-term Memory
Zihong He, Weizhe Lin, Hao Zheng +7
With the rapid advancement of AI systems, their abilities to store, retrieve, and utilize information over the long term - referred to as long-term memory - have become increasingl…
Why you don't overfit, and don't need Bayes if you only train for one epoch
Laurence Aitchison
Here, we show that in the data-rich setting where you only train on each datapoint once (or equivalently, you only train for one epoch), standard "maximum likelihood" training opti…