6 citations · 7 across the 7 of their papers we have counts for
6 papers · 1 filter
Why you don't overfit, and don't need Bayes if you only train for one epoch
Laurence Aitchison
Here, we show that in the data-rich setting where you only train on each datapoint once (or equivalently, you only train for one epoch), standard "maximum likelihood" training opti…
Residual Stream Analysis with Multi-Layer SAEs
Tim Lawson, Lucy Farnik, Conor Houghton +1
Sparse autoencoders (SAEs) are a promising approach to interpreting the internal representations of transformer language models. However, SAEs are usually trained separately on eac…
Using Neural Networks for Data Cleaning in Weather Datasets
Jack R. P. Hanslope, Laurence Aitchison
In climate science, we often want to compare across different datasets. Difficulties can arise in doing this due to inevitable mismatches that arise between observational and reana…
How to set AdamW's weight decay as you scale model and dataset size
Xi Wang, Laurence Aitchison
The scaling of the optimal AdamW weight decay hyperparameter with model and dataset size is critical as we seek to build larger models, but is poorly understood. We show that weigh…
Batch size invariant Adam
Xi Wang, Laurence Aitchison
We propose a batch size invariant version of Adam, for use in large-scale, distributed settings, in which the mini-batch is divided into micro-batches which are distributed among w…
LoRA ensembles for large language model fine-tuning
Xi Wang, Laurence Aitchison, Maja Rudolph
Finetuned LLMs often exhibit poor uncertainty quantification, manifesting as overconfidence, poor calibration, and unreliable prediction results on test data or out-of-distribution…