Probabilistic symmetries and invariant neural networks

Abstract

Treating neural network inputs and outputs as random variables, we characterize the structure of neural networks that can be used to model data that are invariant or equivariant under the action of a compact group. Much recent research has been devoted to encoding invariance under symmetry transformations into neural network architectures, in an effort to improve the performance of deep neural networks in data-scarce, non-i.i.d., or unsupervised settings. By considering group invariance from the perspective of probabilistic symmetry, we establish a link between functional and probabilistic symmetry, and obtain generative functional representations of probability distributions that are invariant or equivariant under the action of a compact group. Our representations completely characterize the structure of neural networks that can be used to model such distributions and yield a general program for constructing invariant stochastic or deterministic neural networks. We demonstrate that examples from the recent literature are special cases, and develop the details of the general program for exchangeable sequences and arrays.

Revised structure for clarity; fixed minor mistakes; incorporated reviewer feedback for publication

Cited by in corpus (15)

• Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
• Janossy Pooling: Learning Deep Permutation-Invariant Functions for Variable-Size Inputs
• Universal approximations of permutation invariant/equivariant functions by deep neural networks
• On the equivalence between graph isomorphism testing and function approximation with GNNs
• LieTransformer: Equivariant self-attention for Lie Groups
• Explainable machine learning of the underlying physics of high-energy particle collisions
• not-MIWAE: Deep Generative Modelling with Missing not at Random Data
• Meta-Learning Stationary Stochastic Process Prediction with Convolutional Neural Processes
• Provably Strict Generalisation Benefit for Equivariant Models
• MetaFun: Meta-Learning with Iterative Functional Updates
• Neural Clustering Processes
• Reconstruction for Powerful Graph Representations
• Equivariant Entity-Relationship Networks
• Robustness through Data Augmentation Loss Consistency
• Properties from Mechanisms: An Equivariance Perspective on Identifiable Representation Learning