Accelerating Transformer Pre-training with 2:4 Sparsity

Yuezhou Hu, Kang Zhao, Weiyu Huang +2

Training large transformers is slow, but recent innovations on GPU architecture give us an advantage. NVIDIA Ampere GPUs can execute a fine-grained 2:4 sparse matrix multiplication…