10 citations · 14 across the 8 of their papers we have counts for
10 papers
CRT-HMAR: Causal Requirement Tracing-Guided Hierarchical Multi-Agent Regulation for Open-Task-Aware Infrared-Visible Image Fusion
Zengyi Yang, Shuai Yuan, Zhong-Cheng Wu +3
Infrared and visible (IR-VIS) image fusion integrates complementary multimodal information into a single fused image to support downstream vision tasks. However, existing methods a…
ZAYA1-8B Technical Report
Robert Washbourne, Rishi Iyer, Tomas Figliolia +15
We present ZAYA1-8B, a reasoning-focused mixture-of-experts (MoE) model with 700M active and 8B total parameters, built on Zyphra's MoE++ architecture. ZAYA1-8B's core pretraining,…
Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
Quentin Anthony, Yury Tokpanov, Skyler Szot +18
We report on the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs and Pollara networking. We distill practical guidance…
Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space
Tomas Figliolia, Nicholas Alonso, Rishi Iyer +2
Multi-headed Attention's (MHA) quadratic compute and linearly growing KV-cache make long-context transformers expensive to train and serve. Prior works such as Grouped Query Attent…
The Zamba2 Suite: Technical Report
Paolo Glorioso, Quentin Anthony, Yury Tokpanov +5
In this technical report, we present the Zamba2 series -- a suite of 1.2B, 2.7B, and 7.4B parameter hybrid Mamba2-transformer models that achieve state of the art performance again…
Zyda-2: a 5 Trillion Token High-Quality Dataset
Yury Tokpanov, Paolo Glorioso, Quentin Anthony +1
In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-t…
Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters
Vasudev Shyam, Jonathan Pilault, Emily Shepperd +2
Our formulation reveals that the reduction across the sequence axis can be efficiently computed in parallel through a tree reduction. Our algorithm, called Tree Attention, for para…
Zyda: A 1.3T Dataset for Open Language Modeling
Yury Tokpanov, Beren Millidge, Paolo Glorioso +4
The size of large language models (LLMs) has scaled dramatically in recent years and their computational and data requirements have surged correspondingly. State-of-the-art languag…
Zamba: A Compact 7B SSM Hybrid Model
Paolo Glorioso, Quentin Anthony, Yury Tokpanov +4
In this technical report, we present Zamba, a novel 7B SSM-transformer hybrid model which achieves competitive performance against leading open-weight models at a comparable scale.…
BlackMamba: Mixture of Experts for State-Space Models
Quentin Anthony, Yury Tokpanov, Paolo Glorioso +1
State-space models (SSMs) have recently demonstrated competitive performance to transformers at large-scale language modeling benchmarks while achieving linear time and memory comp…