6 citations · 7 across the 7 of their papers we have counts for
6 papers · 1 filter
Why you don't overfit, and don't need Bayes if you only train for one epoch
Laurence Aitchison
Here, we show that in the data-rich setting where you only train on each datapoint once (or equivalently, you only train for one epoch), standard "maximum likelihood" training opti…
Human-inspired Perspectives: A Survey on AI Long-term Memory
Zihong He, Weizhe Lin, Hao Zheng +7
With the rapid advancement of AI systems, their abilities to store, retrieve, and utilize information over the long term - referred to as long-term memory - have become increasingl…
Residual Stream Analysis with Multi-Layer SAEs
Tim Lawson, Lucy Farnik, Conor Houghton +1
Sparse autoencoders (SAEs) are a promising approach to interpreting the internal representations of transformer language models. However, SAEs are usually trained separately on eac…
Using Neural Networks for Data Cleaning in Weather Datasets
Jack R. P. Hanslope, Laurence Aitchison
In climate science, we often want to compare across different datasets. Difficulties can arise in doing this due to inevitable mismatches that arise between observational and reana…
How to set AdamW's weight decay as you scale model and dataset size
Xi Wang, Laurence Aitchison
The scaling of the optimal AdamW weight decay hyperparameter with model and dataset size is critical as we seek to build larger models, but is poorly understood. We show that weigh…
Batch size invariant Adam
Xi Wang, Laurence Aitchison
We propose a batch size invariant version of Adam, for use in large-scale, distributed settings, in which the mini-batch is divided into micro-batches which are distributed among w…