Reducing the Cost of Dropout in Flash-Attention by Hiding RNG with GEMM

Haiyue Ma, Jian Liu, Ronny Krashinsky

Dropout, a network operator, when enabled is likely to dramatically impact the performance of Flash-Attention, which in turn increases the end-to-end training time of Large-Languag…