Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models

Bo Gao, Michael W. Spratling, Letizia Gionfrida

Large language models have achieved remarkable success in recent years, primarily due to self-attention. However, traditional Softmax attention suffers from numerical instability a…