The Convolution Exponential and Generalized Sylvester Flows
arXiv:2006.01910
Abstract
This paper introduces a new method to build linear flows, by taking the exponential of a linear transformation. This linear transformation does not need to be invertible itself, and the exponential has the following desirable properties: it is guaranteed to be invertible, its inverse is straightforward to compute and the log Jacobian determinant is equal to the trace of the linear transformation. An important insight is that the exponential can be computed implicitly, which allows the use of convolutional layers. Using this insight, we develop new invertible transformations named convolution exponentials and graph convolution exponentials, which retain the equivariance of their underlying transformations. In addition, we generalize Sylvester Flows and propose Convolutional Sylvester Flows which are based on the generalization and the convolution exponential as basis change. Empirically, we show that the convolution exponential outperforms other linear transformations in generative flows on CIFAR10 and the graph convolution exponential improves the performance of graph normalizing flows. In addition, we show that Convolutional Sylvester Flows improve performance over residual flows as a generative flow model measured in log-likelihood.
arXiv commentAccepted to Neural Information Processing Systems (NeurIPS) 2020
References in corpus (21)
- Auto-Encoding Variational Bayes
- Semi-Supervised Classification with Graph Convolutional Networks
- Spectral Normalization for Generative Adversarial Networks
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- Group Equivariant Convolutional Networks
- Normalizing Flows for Probabilistic Modeling and Inference
- MADE: Masked Autoencoder for Distribution Estimation
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
- Residual Flows for Invertible Generative Modeling
- Exploiting Cyclic Symmetry in Convolutional Neural Networks
- Invariant and Equivariant Graph Networks
- Improving Variational Auto-Encoders using Householder Flow
- Equivariant Flows: sampling configurations for multi-body systems with symmetric energies
- Emerging Convolutions for Generative Normalizing Flows
- Equivariant Hamiltonian Flows
- Graph Normalizing Flows
- Sum-of-Squares Polynomial Flow
- Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
- Cubic-Spline Flows
- Predictive Sampling with Forecasting Autoregressive Models