6 papers · 1 filter
The Optimization Landscape of Learning Compacted Context Models
Thomas Villeneuve, Alex Sandomirsky, Charles O'Neill +2
Many works approach continual learning through the lens of infinite context windows. As an agent puts more observation into context (concretely the KV cache), compacting said conte…
Post-Training Science for Supervised Fine-Tuning
Charles O'Neill, Mudith Jayasekara, Harry Partridge
Every supervised fine-tuning run forces the same chain of decisions, such as learning rate, batch size, LoRA or full fine-tuning, how many epochs, which optimiser, and what data to…
Still: Amortized KV Cache Compaction in a Single Forward Pass
Charles O'Neill, Alex Sandomirsky, Harry Partridge +2
The KV cache is the memory bottleneck of long-horizon language model deployment. Practically, a deployable compactor must be lightweight enough to call during inference, expressive…
DiscoGen: Procedural Generation of Algorithm Discovery Tasks in Machine Learning
Alexander D. Goldie, Zilin Wang, Adrian Hayler +17
Automating the development of machine learning algorithms has the potential to unlock new breakthroughs. However, our ability to improve and evaluate algorithm discovery systems ha…
Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders
Charles O'Neill, Mudith Jayasekara, Max Kirkby
Sparse autoencoders (SAEs) decompose large language model (LLM) activations into latent features that reveal mechanistic structure. Conventional SAEs train on broad data distributi…
A Single Direction of Truth: An Observer Model's Linear Residual Probe Exposes and Steers Contextual Hallucinations
Charles O'Neill, Slava Chalnev, Chi Chi Zhao +2
Contextual hallucinations -- statements unsupported by given context -- remain a significant challenge in AI. We demonstrate a practical interpretability insight: a generator-agnos…