Demystifying the Communication Characteristics for Distributed Transformer Models

Quentin Anthony, Benjamin Michalowicz, Jacob Hatef +5

Deep learning (DL) models based on the transformer architecture have revolutionized many DL applications such as large language models (LLMs), vision transformers, audio generation…