2 citations · 2 across the 7 of their papers we have counts for
4 papers · 1 filter
How Local Mixing Encodes Relative Position in Global NoPE Attention
Cutter Dawes, Nick Alonso, Tom Figliolia +1
The attention operation is naively position invariant. However, positional information is fundamental to natural language, and therefore a variety of explicit position encodings ha…
Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
Quentin Anthony, Yury Tokpanov, Skyler Szot +18
We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs and Pollara networking. We distill practical guidance…
Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
Tomas Figliolia, Nicholas Alonso, Rishi Iyer +2
Multi-headed Attention's (MHA) quadratic compute and linearly growing KV-cache make long-context transformers expensive to train and serve. Prior works such as Grouped Query Attent…
Toward Conversational Agents with Context and Time Sensitive Long-term Memory
Nick Alonso, Tomás Figliolia, Anthony Ndirango +1
There has recently been growing interest in conversational agents with long-term memory which has led to the rapid development of language models that use retrieval-augmented gener…