On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers

Tianchu Ji, Shraddhan Jain, Michael Ferdman +3

How much information do NLP tasks really need from a transformer's attention mechanism at application-time (inference)? From recent work, we know that there is sparsity in transfor…