Hardware-Efficient Softmax and Layer Normalization with Guaranteed Normalization for Edge Devices

Dawon Choi, Hana Kim, Ji-Hoon Kim

In Transformer models, non-GEMM (non-General Matrix Multiplication) operations -- especially Softmax and Layer Normalization (LayerNorm) -- often dominate hardware cost due to thei…