Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism

Shulun Wang, Bin Liu, Feng Liu

Softmax is widely used in neural networks for multiclass classification, gate structure and attention mechanisms. The statistical assumption that the input is normal distributed su…