Self Normalizing Flows

Abstract

Efficient gradient computation of the Jacobian determinant term is a core problem in many machine learning settings, and especially so in the normalizing flow framework. Most proposed flow models therefore either restrict to a function class with easy evaluation of the Jacobian determinant, or an efficient estimator thereof. However, these restrictions limit the performance of such density models, frequently requiring significant depth to reach desired performance levels. In this work, we propose Self Normalizing Flows, a flexible framework for training normalizing flows by replacing expensive terms in the gradient by learned approximate inverses at each layer. This reduces the computational complexity of each layer's exact update from $\mathcal{O}(D^3)$ to $\mathcal{O}(D^2)$, allowing for the training of flow architectures which were otherwise computationally infeasible, while also providing efficient sampling. We show experimentally that such models are remarkably stable and optimize to similar data likelihood values as their exact gradient counterparts, while training more quickly and surpassing the performance of functionally constrained counterparts.

References in corpus (13)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Density estimation using Real NVP
• Residual Flows for Invertible Generative Modeling
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Putting An End to End-to-End: Gradient-Isolated Learning of Representations
• High-Dimensional Probability Estimation with Deep Density Models
• Emerging Convolutions for Generative Normalizing Flows
• A Theoretical Framework for Target Propagation
• Deriving Differential Target Propagation from Iterating Approximate Inverses
• Relative gradient optimization of the Jacobian term in unsupervised deep learning
• The Convolution Exponential and Generalized Sylvester Flows
• Invertible DenseNets
• Training Invertible Linear Layers through Rank-One Perturbations