1 paper · 1 filter
Zimu Zhao
Standard multi-head attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby le…