Equivariance Through Parameter-Sharing

Abstract

We propose to study equivariance in deep neural networks through parameter symmetries. In particular, given a group $\mathcal{G}$ that acts discretely on the input and output of a standard neural network layer ${\varphi }_W:{\Re }^M\to {\Re }^N$, we show that ${\varphi }_W$ is equivariant with respect to $\mathcal{G}$-action iff $\mathcal{G}$ explains the symmetries of the network parameters $W$. Inspired by this observation, we then propose two parameter-sharing schemes to induce the desirable symmetry on $W$. Our procedures for tying the parameters achieve $\mathcal{G}$-equivariance and, under some conditions on the action of $\mathcal{G}$, they guarantee sensitivity to all other permutation groups outside $\mathcal{G}$.

icml'17

References in corpus (2)

• Counting Belief Propagation
• Markov Chains on Orbits of Permutation Groups

Cited by in corpus (57)

• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data
• Gauge Equivariant Convolutional Networks and the Icosahedral CNN
• Recent advances in applying deep reinforcement learning for flow control: perspectives and future directions
• Layer-wise training convolutional neural networks with smaller filters for human activity recognition using wearable sensors
• Universal Invariant and Equivariant Graph Neural Networks
• Sampling using $SU(N)$ gauge equivariant flows
• Deep Models of Interactions Across Sets
• On Learning Sets of Symmetric Elements
• Learning Based User Scheduling in Reconfigurable Intelligent Surface Assisted Multiuser Downlink
• Scale equivariance in CNNs with vector fields
• Relational Pooling for Graph Representations
• Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network
• A General Theory of Equivariant CNNs on Homogeneous Spaces
• A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups
• Burst Photography for Learning to Enhance Extremely Dark Images
• Meta-Learning Symmetries by Reparameterization
• Automatic Symmetry Discovery with Lie Algebra Convolutional Network
• On the Benefits of Invariance in Neural Networks
• Set2Graph: Learning Graphs From Sets
• Rotation-Equivariant Deep Learning for Diffusion MRI
• On the Universality of Invariant Networks
• A Simple Proof of the Universality of Invariant/Equivariant Graph Neural Networks
• ManifoldNet: A Deep Network Framework for Manifold-valued Data
• Machine learning and invariant theory
• Universal Equivariant Multilayer Perceptrons
• Global Attention Improves Graph Networks Generalization
• Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds
• Provably Strict Generalisation Benefit for Equivariant Models
• Quantum algorithms for group convolution, cross-correlation, and equivariant transformations
• On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis
• Solving Continual Combinatorial Selection via Deep Reinforcement Learning
• Neural Clustering Processes
• A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation
• Learning Context-Dependent Choice Functions
• An Overview on the Application of Graph Neural Networks in Wireless Networks
• A Computationally Efficient Neural Network Invariant to the Action of Symmetry Subgroups
• The general theory of permutation equivarant neural networks and higher order graph variational encoders
• Equivariant Entity-Relationship Networks
• Graph Neural Networks for Scalable Radio Resource Management: Architecture Design and Theoretical Analysis
• $\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space
• Deep Architectures for Learning Context-dependent Ranking Functions
• Group Equivariant Stand-Alone Self-Attention For Vision
• Circular-Symmetric Correlation Layer based on FFT
• Adversarial Monte Carlo Meta-Learning of Optimal Prediction Procedures
• Learning Permutation Invariant Representations using Memory Networks
• Equivariant Networks for Pixelized Spheres
• Equivariant Deep Dynamical Model for Motion Prediction
• Training or Architecture? How to Incorporate Invariance in Neural Networks
• Provably Strict Generalisation Benefit for Invariance in Kernel Methods
• Autoequivariant Network Search via Group Decomposition
• Learning Power Control for Cellular Systems with Heterogeneous Graph Neural Network
• Knowledge as Invariance -- History and Perspectives of Knowledge-augmented Machine Learning
• Burst Denoising of Dark Images
• Deep Reinforcement Learning with Symmetric Prior for Predictive Power Allocation to Mobile Users
• Rotation Equivariant 3D Hand Mesh Generation from a Single RGB Image
• Distribution-Based Invariant Deep Networks for Learning Meta-Features