On Infinite-Width Hypernetworks
arXiv:2003.12193
Abstract
{\em Hypernetworks} are architectures that produce the weights of a task-specific {\em primary network}. A notable application of hypernetworks in the recent literature involves learning to output functional representations. In these scenarios, the hypernetwork learns a representation corresponding to the weights of a shallow MLP, which typically encodes shape or image information. While such representations have seen considerable success in practice, they remain lacking in the theoretical guarantees in the wide regime of the standard architectures. In this work, we study wide over-parameterized hypernetworks. We show that unlike typical architectures, infinitely wide hypernetworks do not guarantee convergence to a global minima under gradient descent. We further show that convexity can be achieved by increasing the dimensionality of the hypernetwork's output, to represent wide MLPs. In the dually infinite-width regime, we identify the functional priors of these architectures by deriving their corresponding GP and NTK kernels, the latter of which we refer to as the {\em hyperkernel}. As part of this study, we make a mathematical contribution by deriving tight bounds on high order Taylor expansion terms of standard fully connected ReLU networks.
The first two authors contributed equally
References in corpus (18)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- Deep Neural Networks as Gaussian Processes
- On Exact Computation with an Infinitely Wide Neural Net
- Pay Less Attention with Lightweight and Dynamic Convolutions
- Learning feed-forward one-shot learners
- L2 Regularization versus Batch and Weight Normalization
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
- Stochastic Hyperparameter Optimization through Hypernetworks
- Enhanced Convolutional Neural Tangent Kernels
- Hypernetwork functional image representation
- Kernel and Rich Regimes in Overparametrized Models
- Principled Weight Initialization for Hypernetworks
- Hyper-Graph-Network Decoders for Block Codes
- The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
- Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
- On Random Kernels of Residual Architectures