Forget Attention: Importance-Aware Attention Is All You Need

Suhyeong Shin, Yeongwook Yang

Combining attention's global retrieval with the sequential importance signal of state space models (SSMs) is the open challenge of hybrid language modeling. Transformers see everyw…