A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Experts

Viet Nguyen, Tuan Minh Pham, Thinh Cao +4

Self-attention has greatly contributed to the success of the widely used Transformer architecture by enabling learning from data with long-range dependencies. In an effort to impro…