most citedOn the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers

2 citations · 2 across the 1 of their papers we have counts for

collaborators

1 paper

cs.CL2021★ 2 cited

On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers

Tianchu Ji, Shraddhan Jain, Michael Ferdman +3

How much information do NLP tasks really need from a transformer's attention mechanism at application-time (inference)? From recent work, we know that there is sparsity in transfor…