Is Attention Better Than Matrix Decomposition?

Abstract

As an essential ingredient of modern deep learning, attention mechanism, especially self-attention, plays a vital role in the global correlation discovery. However, is hand-crafted attention irreplaceable when modeling the global context? Our intriguing finding is that self-attention is not better than the matrix decomposition (MD) model developed 20 years ago regarding the performance and computational cost for encoding the long-distance dependencies. We model the global context issue as a low-rank recovery problem and show that its optimization algorithms can help design global information blocks. This paper then proposes a series of Hamburgers, in which we employ the optimization algorithms for solving MDs to factorize the input representations into sub-matrices and reconstruct a low-rank embedding. Hamburgers with different MDs can perform favorably against the popular global context module self-attention when carefully coping with gradients back-propagated through MDs. Comprehensive experiments are conducted in the vision tasks where it is crucial to learn the global context, including semantic segmentation and image generation, demonstrating significant improvements over self-attention and its variants.

ICLR 2021

References in corpus (10)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• A Structured Self-attentive Sentence Embedding
• Recurrent Models of Visual Attention
• Linformer: Self-Attention with Linear Complexity
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View
• High-Fidelity Image Generation With Fewer Labels
• LatentGNN: Learning Efficient Non-local Relations for Visual Recognition

Cited by in corpus (11)

• A Survey on Visual Transformer
• Attention Mechanisms in Computer Vision: A Survey
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• SOFT: Softmax-free Transformer with Linear Complexity
• Factorizer: A Scalable Interpretable Approach to Context Modeling for Medical Image Segmentation
• On Training Implicit Models
• VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization
• TransLoc3D : Point Cloud based Large-scale Place Recognition using Adaptive Receptive Fields
• Sampling Equivariant Self-attention Networks for Object Detection in Aerial Images
• One-step differentiation of iterative algorithms
• Hierarchical Pyramid Representations for Semantic Segmentation