3 papers
cs.LG2026
Online Vector Quantized Attention
Nick Alonso, Tomas Figliolia, Beren Millidge
Standard sequence mixing layers used in language models struggle to balance efficiency and performance. Self-attention performs well on long context tasks but has expensive quadrat…
cs.CL2026
Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
Tomas Figliolia, Nicholas Alonso, Rishi Iyer +2
Multi-headed Attention's (MHA) quadratic compute and linearly growing KV-cache make long-context transformers expensive to train and serve. Prior works such as Grouped Query Attent…
cs.IR2024
Mixture-of-PageRanks: Replacing Long-Context with Real-Time, Sparse GraphRAG
Nicholas Alonso, Beren Millidge
Recent advances have extended the context window of frontier LLMs dramatically, from a few thousand tokens up to millions, enabling entire books and codebases to fit into context.…