Provably learning a multi-head attention layer

Sitan Chen, Yuanzhi Li

The multi-head attention layer is one of the key components of the transformer architecture that sets it apart from traditional feed-forward models. Given a sequence length $k$, at…