Centroid Transformers: Learning to Abstract with Attention

Abstract

Self-attention, as the key block of transformers, is a powerful mechanism for extracting features from the inputs. In essence, what self-attention does is to infer the pairwise relations between the elements of the inputs, and modify the inputs by propagating information between input pairs. As a result, it maps inputs to N outputs and casts a quadratic $O(N^2)$ memory and time complexity. We propose centroid attention, a generalization of self-attention that maps N inputs to M outputs $(M\le N)$, such that the key information in the inputs are summarized in the smaller number of outputs (called centroids). We design centroid attention by amortizing the gradient descent update rule of a clustering objective function on the inputs, which reveals an underlying connection between attention and clustering. By compressing the inputs to the centroids, we extract the key information useful for prediction and also reduce the computation of the attention module and the subsequent layers. We apply our method to various applications, including abstractive text summarization, 3D vision, and image processing. Empirical results demonstrate the effectiveness of our method over the standard transformers.

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Striving for Simplicity: The All Convolutional Net
• Longformer: The Long-Document Transformer
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
• Linformer: Self-Attention with Linear Complexity
• Towards K-means-friendly Spaces: Simultaneous Deep Learning and Clustering
• Reformer: The Efficient Transformer
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Clustering with Deep Learning: Taxonomy and New Methods
• Language Modeling with Deep Transformers
• Learning Deep Transformer Models for Machine Translation
• Dynamic Routing Between Capsules
• Hopfield Networks is All You Need
• PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
• Deeper or Wider Networks of Point Clouds with Self-attention?

Cited by in corpus (3)

• HeadPosr: End-to-end Trainable Head Pose Estimation using Transformer Encoders
• Token Pooling in Vision Transformers
• T-EMDE: Sketching-based global similarity for cross-modal retrieval