4 papers · 1 filter
The Optimization Landscape of Learning Compacted Context Models
Thomas Villeneuve, Alex Sandomirsky, Charles O'Neill +2
Many works approach continual learning through the lens of infinite context windows. As an agent puts more observation into context (concretely the KV cache), compacting said conte…
Still: Amortized KV Cache Compaction in a Single Forward Pass
Charles O'Neill, Alex Sandomirsky, Harry Partridge +2
The KV cache is the memory bottleneck of long-horizon language model deployment. Practically, a deployable compactor must be lightweight enough to call during inference, expressive…
Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders
Charles O'Neill, Mudith Jayasekara, Max Kirkby
Sparse autoencoders (SAEs) decompose large language model (LLM) activations into latent features that reveal mechanistic structure. Conventional SAEs train on broad data distributi…
A Single Direction of Truth: An Observer Model's Linear Residual Probe Exposes and Steers Contextual Hallucinations
Charles O'Neill, Slava Chalnev, Chi Chi Zhao +2
Contextual hallucinations -- statements unsupported by given context -- remain a significant challenge in AI. We demonstrate a practical interpretability insight: a generator-agnos…