Differential Transformer

Tianzhu Ye, Li Dong, Yuqing Xia +4

Transformer tends to overallocate attention to irrelevant context. In this work, we introduce Diff Transformer, which amplifies attention to the relevant context while canceling no…