ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters

Shiwei Liu, Guanchen Tao, Yifei Zou +7

The self-attention mechanism distinguishes transformer-based large language models (LLMs) apart from convolutional and recurrent neural networks. Despite the performance improvemen…