most citedAnalyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

70 citations · 70 across the 2 of their papers we have counts for

collaborators

2 papers

cs.CL2019★ 70 cited

Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned

Elena Voita, David Talbot, Fedor Moiseev +2

Multi-head self-attention is a key component of the Transformer, a state-of-the-art architecture for neural machine translation. In this work we evaluate the contribution made by i…

cs.DS2007

Bloom maps

David Talbot, John Talbot

We consider the problem of succinctly encoding a static map to support approximate queries. We derive upper and lower bounds on the space requirements in terms of the error rate an…