AttentionX: Exploiting Consensus Discrepancy In Attention from A Distributed Optimization Perspective

Guoqiang Zhang, Richard Heusdens

In this paper, we extend the standard Attention in transformer by exploiting the consensus discrepancy from a distributed optimization perspective, referred to as AttentionX. It is…