NanoFlow: Scalable Normalizing Flows with Sublinear Parameter Complexity
arXiv:2006.06280
Abstract
Normalizing flows (NFs) have become a prominent method for deep generative models that allow for an analytic probability density estimation and efficient synthesis. However, a flow-based network is considered to be inefficient in parameter complexity because of reduced expressiveness of bijective mapping, which renders the models unfeasibly expensive in terms of parameters. We present an alternative parameterization scheme called NanoFlow, which uses a single neural density estimator to model multiple transformation stages. Hence, we propose an efficient parameter decomposition method and the concept of flow indication embedding, which are key missing components that enable density estimation from a single neural network. Experiments performed on audio and image models confirm that our method provides a new parameter-efficient solution for scalable NFs with significant sublinear parameter complexity.
NeurIPS 2020. 13 pages, 3 figures, 6 tables. Code: https://github.com/L0SG/NanoFlow
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- WaveNet: A Generative Model for Raw Audio
- Pixel Recurrent Neural Networks
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Image Transformer
- Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
- Residual Flows for Invertible Generative Modeling
- How to train your neural ODE: the world of Jacobian and kinetic regularization
- WaveFlow: A Compact Flow-based Model for Raw Audio
- Emerging Convolutions for Generative Normalizing Flows
- TraDE: Transformers for Density Estimation