Shared DIFF Transformer

Yueyang Cang, Yuhang Liu, Xiaoteng Zhang +2

DIFF Transformer improves attention allocation by enhancing focus on relevant context while suppressing noise. It introduces a differential attention mechanism that calculates the…