Discrete Cosine Transform Based Decorrelated Attention for Vision Transformers

Hongyi Pan, Emadeldeen Hamdan, Xin Zhu +2

Self-attention is central to the success of Transformer architectures; however, learning the query, key, and value projections from random initialization remains challenging and co…