SparkAttention: High-Performance Multi-Head Attention for Large Models on Volta GPU Architecture

Youxuan Xu, Tong Wu, Shigang Li +2

Transformer are widely used in various fields such as natural language processing and computer vision. However, the training time for large Transformer models can be challenging du…