Group Equivariant Stand-Alone Self-Attention For Vision
arXiv:2010.00977
Abstract
We provide a general self-attention formulation to impose group equivariance to arbitrary symmetry groups. This is achieved by defining positional encodings that are invariant to the action of the group considered. Since the group acts on the positional encoding directly, group equivariant self-attention networks (GSA-Nets) are steerable by nature. Our experiments on vision benchmarks demonstrate consistent improvements of GSA-Nets over non-equivariant self-attention networks.
arXiv commentProceedings of the 9th International Conference on Learning Representations (ICLR), 2021
References in corpus (30)
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Linformer: Self-Attention with Linear Complexity
- Searching for Activation Functions
- Group Equivariant Convolutional Networks
- Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
- SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
- Big Bird: Transformers for Longer Sequences
- On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
- 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data
- Rotation equivariant vector field networks
- Stand-Alone Self-Attention in Vision Models
- Long Short-Term Memory-Networks for Machine Reading
- Gauge Equivariant Convolutional Networks and the Icosahedral CNN
- Rethinking Attention with Performers
- Equivariance Through Parameter-Sharing
- General -Equivariant Steerable CNNs
- Group Equivariant Capsule Networks
- Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
- On Learning Sets of Symmetric Elements
- Deep Scale-spaces: Equivariance Over Scale
- Scale equivariance in CNNs with vector fields
- LieTransformer: Equivariant self-attention for Lie Groups
- Equivariant Transformer Networks
- Attentive Group Equivariant Convolutional Networks
- Understanding the Difficulty of Training Transformers
- Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring In Data
- DeepSphere: a graph-based spherical CNN
- RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks
- Universal Equivariant Multilayer Perceptrons
- Cross-Domain 3D Equivariant Image Embeddings