2 citations · 2 across the 4 of their papers we have counts for
3 papers · 1 filter
How Local Mixing Encodes Relative Position in Global NoPE Attention
Cutter Dawes, Nick Alonso, Tom Figliolia +1
The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings ha…
Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
Tomas Figliolia, Nicholas Alonso, Rishi Iyer +2
Multi-headed Attention's (MHA) quadratic compute and linearly growing KV-cache make long-context transformers expensive to train and serve. Prior works such as Grouped Query Attent…
Toward Conversational Agents with Context and Time Sensitive Long-term Memory
Nick Alonso, Tomás Figliolia, Anthony Ndirango +1
There has recently been growing interest in conversational agents with long-term memory which has led to the rapid development of language models that use retrieval-augmented gener…