Towards understanding how attention mechanism works in deep learning

Tianyu Ruan, Shihua Zhang

Attention mechanism has been extensively integrated within mainstream neural network architectures, such as Transformers and graph attention networks. Yet, its underlying working p…