Densely connected normalizing flows

Abstract

Normalizing flows are bijective mappings between inputs and latent representations with a fully factorized distribution. They are very attractive due to exact likelihood valuation and efficient sampling. However, their effective capacity is often insufficient since the bijectivity constraint limits the model width. We address this issue by incrementally padding intermediate representations with noise. We precondition the noise in accordance with previous invertible units, which we describe as cross-unit coupling. Our invertible glow-like modules increase the model expressivity by fusing a densely connected block with Nystrom self-attention. We refer to our architecture as DenseFlow since both cross-unit and intra-module couplings rely on dense connectivity. Experiments show significant improvements due to the proposed contributions and reveal state-of-the-art density estimation under moderate computing budgets.

arXiv commentAccepted at NeurIPS2021

References in corpus (24)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• Improved Training of Wasserstein GANs
• Variational Inference with Normalizing Flows
• A Simple Neural Attentive Meta-Learner
• Improved Denoising Diffusion Probabilistic Models
• Variational Diffusion Models
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Residual Flows for Invertible Generative Modeling
• Improving Variational Inference with Inverse Autoregressive Flow
• Quantifying the Carbon Emissions of Machine Learning
• Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
• Hybrid Models with Deep and Invertible Features
• Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
• PixelVAE++: Improved PixelVAE with Discrete Prior
• Consistency Regularization for Variational Auto-Encoders
• Invertible DenseNets with Concatenated LipSwish
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Adaptive Density Estimation for Generative Models
• Invertible DenseNets

Cited by in corpus (3)

• normflows: A PyTorch Package for Normalizing Flows
• Hybrid Open-set Segmentation with Synthetic Negative Data
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding