Invertible DenseNets with Concatenated LipSwish
arXiv:2102.02694
Abstract
We introduce Invertible Dense Networks (i-DenseNets), a more parameter efficient extension of Residual Flows. The method relies on an analysis of the Lipschitz continuity of the concatenation in DenseNets, where we enforce invertibility of the network by satisfying the Lipschitz constant. Furthermore, we propose a learnable weighted concatenation, which not only improves the model performance but also indicates the importance of the concatenated weighted representation. Additionally, we introduce the Concatenated LipSwish as activation function, for which we show how to enforce the Lipschitz condition and which boosts performance. The new architecture, i-DenseNet, out-performs Residual Flow and other flow-based models on density estimation evaluated in bits per dimension, where we utilize an equal parameter budget. Moreover, we show that the proposed model out-performs Residual Flows when trained as a hybrid model where the model is both a generative and a discriminative model.
arXiv commentAccepted at Neural Information Processing Systems (NeurIPS) 2021. This is an extension of Invertible DenseNets (arXiv:2010.02125). arXiv admin note: text overlap with arXiv:2010.02125
References in corpus (17)
- Spectral Normalization for Generative Adversarial Networks
- Pixel Recurrent Neural Networks
- NICE: Non-linear Independent Components Estimation
- Density estimation using Real NVP
- A note on the evaluation of generative models
- A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
- Residual Flows for Invertible Generative Modeling
- Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units
- Sylvester Normalizing Flows for Variational Inference
- Improving Variational Auto-Encoders using Householder Flow
- High-Dimensional Probability Estimation with Deep Density Models
- Excessive Invariance Causes Adversarial Vulnerability
- Hybrid Models with Deep and Invertible Features
- Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks
- Learning Discrete Distributions by Dequantization
- The Convolution Exponential and Generalized Sylvester Flows