PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

Abstract

PixelCNNs are a recently proposed class of powerful generative models with tractable likelihood. Here we discuss our implementation of PixelCNNs which we make available at https://github.com/openai/pixel-cnn. Our implementation contains a number of modifications to the original model that both simplify its structure and improve its performance. 1) We use a discretized logistic mixture likelihood on the pixels, rather than a 256-way softmax, which we find to speed up training. 2) We condition on whole pixels, rather than R/G/B sub-pixels, simplifying the model structure. 3) We use downsampling to efficiently capture structure at multiple resolutions. 4) We introduce additional short-cut connections to further speed up optimization. 5) We regularize the model using dropout. Finally, we present state-of-the-art log likelihood results on CIFAR-10 to demonstrate the usefulness of these modifications.

References in corpus (10)

• Auto-Encoding Variational Bayes
• Conditional Image Generation with PixelCNN Decoders
• Wide Residual Networks
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Density estimation using Real NVP
• Neural Machine Translation in Linear Time
• RNADE: The real-valued neural autoregressive density-estimator
• Generative Image Modeling Using Spatial LSTMs
• Mixtures of conditional Gaussian scale mixtures applied to multiscale image representations

Cited by in corpus (21)

• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• How Generative Adversarial Networks and Their Variants Work: An Overview
• Deep Unsupervised Cardinality Estimation
• Generative Probabilistic Novelty Detection with Adversarial Autoencoders
• Towards Deeper Understanding of Variational Autoencoding Models
• MelNet: A Generative Model for Audio in the Frequency Domain
• Generative Models for Automatic Chemical Design
• Nonparametric Density Estimation for High-Dimensional Data - Algorithms and Applications
• Bayesian MRI Reconstruction with Joint Uncertainty Estimation using Diffusion Models
• Discrete Sequential Prediction of Continuous Actions for Deep RL
• 3D-Aware Scene Manipulation via Inverse Graphics
• Dynamical mass inference of galaxy clusters with neural flows
• SARM: Sparse Autoregressive Model for Scalable Generation of Sparse Images in Particle Physics
• WaveCycleGAN2: Time-domain Neural Post-filter for Speech Waveform Generation
• Associative Compression Networks for Representation Learning
• Neural Network Statistical Mechanics
• Learning to Play by Imitating Humans
• Generative Priors for MRI Reconstruction Trained from Magnitude-Only Images Using Phase Augmentation
• iVPF: Numerical Invertible Volume Preserving Flow for Efficient Lossless Compression
• Understanding Classifier Mistakes with Generative Models
• Deep Nonparametric Estimation of Discrete Conditional Distributions via Smoothed Dyadic Partitioning