Learning and Generalization in Overparameterized Normalizing Flows

Abstract

In supervised learning, it is known that overparameterized neural networks with one hidden layer provably and efficiently learn and generalize, when trained using stochastic gradient descent with a sufficiently small learning rate and suitable initialization. In contrast, the benefit of overparameterization in unsupervised learning is not well understood. Normalizing flows (NFs) constitute an important class of models in unsupervised learning for sampling and density estimation. In this paper, we theoretically and empirically analyze these models when the underlying neural network is a one-hidden-layer overparametrized network. Our main contributions are two-fold: (1) On the one hand, we provide theoretical and empirical evidence that for constrained NFs (this class of NFs underlies many NF constructions) with the one-hidden-layer network, overparametrization hurts training. (2) On the other hand, we prove that unconstrained NFs, a recently introduced model, can efficiently learn any reasonable data distribution under minimal assumptions when the underlying network is overparametrized and has one hidden-layer.

75 pages, Accepted in AISTATS 2022

References in corpus (8)

• Auto-Encoding Variational Bayes
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Normalizing Flows: An Introduction and Review of Current Methods
• Improving Variational Auto-Encoders using Householder Flow
• Overparameterized Neural Networks Implement Associative Memory
• Generalization in Deep Networks: The Role of Distance from Initialization
• The Expressive Power of a Class of Normalizing Flow Models
• Understanding Overparameterization in Generative Adversarial Networks