RCStat: A Statistical Framework for using Relative Contextualization in Transformers

Debabrata Mahapatra, Shubham Agarwal, Apoorv Saxena +1

Prior work on input-token importance in auto-regressive transformers has relied on Softmax-normalized attention weights, which obscure the richer structure of pre-Softmax query-key…