ATTENTION2D: Communication Efficient Distributed Self-Attention Mechanism

Venmugil Elango

Transformer-based models have emerged as a leading architecture for natural language processing, natural language generation, and image generation tasks. A fundamental element of t…