A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training

Cheng Huan, Hongfwei Yuan

This paper develops a mean-field theory for a simplified single-layer causal multi-head self-attention model trained by cross-entropy minimization. Each attention head is treated a…