Efficient Attention via Control Variates

Lin Zheng, Jianbo Yuan, Chong Wang +1

Random-feature-based attention (RFA) is an efficient approximation of softmax attention with linear runtime and space complexity. However, the approximation gap between RFA and con…