MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometries

Michele Paolicelli, Alessandro Petruzzelli, Alessandro Franceso Maria Martina +2

The quadratic complexity of dense self-attention remains a central bottleneck for long-context language modeling. Many efficient alternatives address this cost by deciding in advan…