Inductive Biases and Variable Creation in Self-Attention Mechanisms

Benjamin L. Edelman, Surbhi Goel, Sham Kakade +1

Self-attention, an architectural motif designed to model long-range interactions in sequential data, has driven numerous recent breakthroughs in natural language processing and bey…