Hierarchical Shift Mixing -- Beyond Dense Attention in Transformers

Robert Forchheimer

Since the introduction of the Transformer architecture for large language models, the softmax-based attention layer has faced increasing scrutinity due to its quadratic-time comput…