Fast Transformer Decoding: One Write-Head is All You Need

Noam Shazeer

Multi-head attention layers, as used in the Transformer neural sequence model, are a powerful alternative to RNNs for moving information across and between sequences. While trainin…