High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classification

Nicholas Barnfield, Hugo Cui, Yue M. Lu

When and how can an attention mechanism learn to selectively attend to informative tokens, thereby enabling detection of weak, rare, and sparsely located features? We address these…