Asymmetric Attention Heads: Structured Head-Wise Context Allocation for Transformer Attention

Zimu Zhao

Standard multi-head attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby le…