Transformer Approximations from ReLUs

Jerry Yao-Chieh Hu, Mingcheng Lu, Yi-Chen Lee +1

We provide a systematic recipe for translating ReLU approximation results to softmax attention mechanism. This recipe covers many common approximation targets. Importantly, it yiel…