On the Dynamics of Training Attention Models

Haoye Lu, Yongyi Mao, Amiya Nayak

The attention mechanism has been widely used in deep neural networks as a model component. By now, it has become a critical building block in many state-of-the-art natural language…