4 papers
A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
Daniel Barley, Jonathan Leis, Benjamin Klenk +1
Pipeline parallelism is a key technique for distributed training of large language models because it reduces per-device parameter and activation memory. However, comparing pipeline…
Scalable and Efficient Intra- and Inter-node Interconnection Networks for Post-Exascale Supercomputers and Data centers
Joaquin Tarraga-Moreno, Daniel Barley, Francisco J. Andujar Munoz +5
The rapid growth of data-intensive applications such as generative AI, scientific simulations, and large-scale analytics is driving modern supercomputers and data centers toward in…
Less Memory Means smaller GPUs: Backpropagation with Compressed Activations
Daniel Barley, Holger Fröning
The ever-growing scale of deep neural networks (DNNs) has lead to an equally rapid growth in computational resource requirements. Many recent architectures, most prominently Large…
Compressing the Backward Pass of Large-Scale Neural Architectures by Structured Activation Pruning
Daniel Barley, Holger Fröning
The rise of Deep Neural Networks (DNNs) has led to an increase in model size and complexity, straining the memory capacity of GPUs. Sparsity in DNNs, characterized as structural or…