Adaptive Head Budgeting for Efficient Multi-Head Attention

Bilal Faye, Abdoulaye Mbaye, Hanane Azzag +1

Multi-head attention enables Transformers to capture diverse representations, but all attention heads are typically activated for every input, regardless of task complexity. For co…