Autoequivariant Network Search via Group Decomposition

Abstract

Recent works show that group equivariance as an inductive bias improves neural network performance for both classification and generation. However, designing group-equivariant neural networks is challenging when the group of interest is large and is unknown. Moreover, inducing equivariance can significantly reduce the number of independent parameters in a network with fixed feature size, affecting its overall performance. We address these problems by proving a new group-theoretic result in the context of equivariant neural networks that shows that a network is equivariant to a large group if and only if it is equivariant to smaller groups from which it is constructed. Using this result, we design a novel fast group equivariant construction algorithm, and a deep Q-learning-based search algorithm in a reduced search space, yielding what we call autoequivariant networks (AENs). AENs find the right balance between equivariance and network size when tested on new benchmark datasets, G-MNIST and G-Fashion-MNIST, obtained via group transformations on MNIST and Fashion-MNIST respectively that we release. Extending these results to group convolutional neural networks, where we optimize between equivariances, augmentations, and network sizes, we find group equivariance to be the most dominating factor in all high-performing GCNNs on several datasets like CIFAR10, SVHN, RotMNIST, ASL, EMNIST, and KMNIST.

References in corpus (20)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Group Equivariant Convolutional Networks
• Designing Neural Network Architectures using Reinforcement Learning
• Making Convolutional Networks Shift-Invariant Again
• SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• Spherical CNNs
• Universal Invariant and Equivariant Graph Neural Networks
• Equivariance Through Parameter-Sharing
• General $E(2)$-Equivariant Steerable CNNs
• Invariant and Equivariant Graph Networks
• Group Equivariant Capsule Networks
• Lorentz Group Equivariant Neural Network for Particle Physics
• Explorations in Homeomorphic Variational Auto-Encoding
• Deep Scale-spaces: Equivariance Over Scale
• LieTransformer: Equivariant self-attention for Lie Groups
• A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups
• Pretrained AI Models: Performativity, Mobility, and Change
• Co-Attentive Equivariant Neural Networks: Focusing Equivariance On Transformations Co-Occurring In Data