Implicit Bias of Linear Equivariant Networks

Abstract

Group equivariant convolutional neural networks (G-CNNs) are generalizations of convolutional neural networks (CNNs) which excel in a wide range of technical applications by explicitly encoding symmetries, such as rotations and permutations, in their architectures. Although the success of G-CNNs is driven by their \emph{explicit} symmetry bias, a recent line of work has proposed that the \emph{implicit} bias of training algorithms on particular architectures is key to understanding generalization for overparameterized neural nets. In this context, we show that $L$-layer full-width linear G-CNNs trained via gradient descent for binary classification converge to solutions with low-rank Fourier matrix coefficients, regularized by the $2/L$-Schatten matrix norm. Our work strictly generalizes previous analysis on the implicit bias of linear CNNs to linear G-CNNs over all finite groups, including the challenging setting of non-commutative groups (such as permutations), as well as band-limited G-CNNs over infinite groups. We validate our theorems via experiments on a variety of groups, and empirically explore more realistic nonlinear networks, which locally capture similar regularization patterns. Finally, we provide intuitive interpretations of our Fourier space implicit regularization results in real space via uncertainty principles.

arXiv comment21 pages, 19 figures

References in corpus (11)

• Group Equivariant Convolutional Networks
• Deep Neural Networks as Gaussian Processes
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• Characterizing Implicit Bias in Terms of Optimization Geometry
• General $E(2)$-Equivariant Steerable CNNs
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• A novel set of rotationally and translationally invariant features for images based on the non-commutative bispectrum
• On the Benefits of Invariance in Neural Networks
• Provably Strict Generalisation Benefit for Equivariant Models
• Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm
• Generalized moduli of continuity under irregular or random deformations via multiscale analysis