Inductive Bias of Multi-Channel Linear Convolutional Networks with Bounded Weight Norm

Abstract

We provide a function space characterization of the inductive bias resulting from minimizing the ${\ell }_2$ norm of the weights in multi-channel convolutional neural networks with linear activations and empirically test our resulting hypothesis on ReLU networks trained using gradient descent. We define an induced regularizer in the function space as the minimum ${\ell }_2$ norm of weights of a network required to realize a function. For two layer linear convolutional networks with $C$ output channels and kernel size $K$, we show the following: (a) If the inputs to the network are single channeled, the induced regularizer for any $K$ is independent of the number of output channels $C$. Furthermore, we derive the regularizer is a norm given by a semidefinite program (SDP). (b) In contrast, for multi-channel inputs, multiple output channels can be necessary to merely realize all matrix-valued linear functions and thus the inductive bias does depend on $C$. However, for sufficiently large $C$, the induced regularizer is again given by an SDP that is independent of $C$. In particular, the induced regularizer for $K=1$ and $K=D$ (input dimension) is given in closed form as the nuclear norm and the ${\ell }_{2,1}$ group-sparse norm, respectively, of the Fourier coefficients of the linear predictor. We investigate the broader applicability of our theoretical results to implicit regularization from gradient descent on linear and ReLU networks through experiments on MNIST and CIFAR-10 datasets.

arXiv commentAppeared at COLT 2022

References in corpus (8)

• Characterizing Implicit Bias in Terms of Optimization Geometry
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel
• Implicit Regularization in Deep Learning May Not Be Explainable by Norms
• Risk and parameter convergence of logistic regression
• Identity Crisis: Memorization and Generalization under Extreme Overparameterization
• Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models

Cited by in corpus (6)

• A Geometric Analysis of Neural Collapse with Unconstrained Features
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks
• On Margin Maximization in Linear and ReLU Networks
• Implicit Bias of Linear Equivariant Networks
• The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer Linear Networks