Woodbury Transformations for Deep Generative Flows

Abstract

Normalizing flows are deep generative models that allow efficient likelihood calculation and sampling. The core requirement for this advantage is that they are constructed using functions that can be efficiently inverted and for which the determinant of the function's Jacobian can be efficiently computed. Researchers have introduced various such flow operations, but few of these allow rich interactions among variables without incurring significant computational costs. In this paper, we introduce Woodbury transformations, which achieve efficient invertibility via the Woodbury matrix identity and efficient determinant calculation via Sylvester's determinant identity. In contrast with other operations used in state-of-the-art normalizing flows, Woodbury transformations enable (1) high-dimensional interactions, (2) efficient sampling, and (3) efficient likelihood evaluation. Other similar operations, such as 1x1 convolutions, emerging convolutions, or periodic convolutions allow at most two of these three advantages. In our experiments on multiple image datasets, we find that Woodbury transformations allow learning of higher-likelihood models than other flow architectures while still enjoying their efficiency advantages.

arXiv commentAccepted to NeurIPS 2020

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• WaveNet: A Generative Model for Raw Audio
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Pixel Recurrent Neural Networks
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• NICE: Non-linear Independent Components Estimation
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Density estimation using Real NVP
• MADE: Masked Autoencoder for Distribution Estimation
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Residual Flows for Invertible Generative Modeling
• Invertible Residual Networks
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• Sylvester Normalizing Flows for Variational Inference
• Discrete Flows: Invertible Generative Models of Discrete Data
• Neural Spline Flows

Cited by in corpus (1)

• Scalable Learning and MAP Inference for Nonsymmetric Determinantal Point Processes