Convolution-enhanced Evolving Attention Networks

Abstract

Attention-based neural networks, such as Transformers, have become ubiquitous in numerous applications, including computer vision, natural language processing, and time-series analysis. In all kinds of attention networks, the attention maps are crucial as they encode semantic dependencies between input tokens. However, most existing attention networks perform modeling or reasoning based on representations , wherein the attention maps of different layers are learned separately without explicit interactions. In this paper, we propose a novel and generic evolving attention mechanism, which directly models the evolution of inter-token relationships through a chain of residual convolutional modules. The major motivations are twofold. On the one hand, the attention maps in different layers share transferable knowledge, thus adding a residual connection can facilitate the information flow of inter-token relationships across layers. On the other hand, there is naturally an evolutionary trend among attention maps at different abstraction levels, so it is beneficial to exploit a dedicated convolution-based module to capture this process. Equipped with the proposed mechanism, the convolution-enhanced evolving attention networks achieve superior performance in various applications, including time-series representation, natural language understanding, machine translation, and image classification. Especially on time-series representation tasks, Evolving Attention-enhanced Dilated Convolutional (EA-DC-) Transformer outperforms state-of-the-art models significantly, achieving an average of 17% improvement compared to the best SOTA. To the best of our knowledge, this is the first work that explicitly models the layer-wise evolution of attention maps. Our implementation is available at https://github.com/pkuyym/EvolvingAttention.

arXiv commentAccepted by IEEE T-PAMI. Extension of the previous work (arXiv:2102.12895). arXiv admin note: text overlap with arXiv:2102.12895

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Neural Machine Translation by Jointly Learning to Align and Translate
• WaveNet: A Generative Model for Raw Audio
• Self-Attention Generative Adversarial Networks
• PCT: Point cloud transformer
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• On the Variance of the Adaptive Learning Rate and Beyond
• Attention is not Explanation
• Unsupervised Scalable Representation Learning for Multivariate Time Series
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Stand-Alone Self-Attention in Vision Models
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Lite Transformer with Long-Short Range Attention
• Sparse Sinkhorn Attention
• PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentation
• Poolingformer: Long Document Modeling with Pooling Attention
• Improving Clinical Predictions through Unsupervised Time Series Representation Learning
• Towards Efficient Scene Understanding via Squeeze Reasoning
• EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm
• Monash University, UEA, UCR Time Series Extrinsic Regression Archive
• Evolving Attention with Residual Convolutions
• Monarch: Expressive Structured Matrices for Efficient and Accurate Training