From the 1 of 24 linked papers with an AI index.
5 papers · 1 filter
Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
Tomas Figliolia, Nicholas Alonso, Rishi Iyer +2
Multi-headed Attention's (MHA) quadratic compute and linearly growing KV-cache make long-context transformers expensive to train and serve. Prior works such as Grouped Query Attent…
Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
Quentin Anthony, Yury Tokpanov, Skyler Szot +18
We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs and Pollara networking. We distill practical guidance…
Zyda-2: a 5 Trillion Token High-Quality Dataset
Yury Tokpanov, Paolo Glorioso, Quentin Anthony +1
In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-t…
Zyda: A 1.3T Dataset for Open Language Modeling
Yury Tokpanov, Beren Millidge, Paolo Glorioso +4
The size of large language models (LLMs) has scaled dramatically in recent years and their computational and data requirements have surged correspondingly. State-of-the-art languag…
Toward Conversational Agents with Context and Time Sensitive Long-term Memory
Nick Alonso, Tomás Figliolia, Anthony Ndirango +1
There has recently been growing interest in conversational agents with long-term memory which has led to the rapid development of language models that use retrieval-augmented gener…