1 citations · 1 across the 3 of their papers we have counts for
3 papers
Scaling Large Language Model Training on Frontier with Low-Bandwidth Partitioning
Lang Xu, Quentin Anthony, Jacob Hatef +4
Scaling up Large Language Model(LLM) training involves fitting a tremendous amount of training parameters across a limited number of workers. However, methods like ZeRO-3 that dras…
Demystifying the Communication Characteristics for Distributed Transformer Models
Quentin Anthony, Benjamin Michalowicz, Jacob Hatef +5
Deep learning (DL) models based on the transformer architecture have revolutionized many DL applications such as large language models (LLMs), vision transformers, audio generation…
The Case for Co-Designing Model Architectures with Hardware
Quentin Anthony, Jacob Hatef, Deepak Narayanan +6
While GPUs are responsible for training the vast majority of state-of-the-art deep learning models, the implications of their architecture are often overlooked when designing new d…