Understanding Differential Transformer Unchains Pretrained Self-Attentions

Chaerin Kong, Jiho Jang, Nojun Kwak

Differential Transformer has recently gained significant attention for its impressive empirical performance, often attributed to its ability to perform noise canceled attention. Ho…