83.4k citations · 107.7k across the 29 of their papers we have counts for
30 papers
DeMo: Decoupled Momentum Optimization
Bowen Peng, Lizhang Chen, Baiyu Su +3
Scaling neural network training increasingly depends on synchronous data-parallelism, yet full-precision gradient all-reduce imposes a severe communication bottleneck. We propose D…
EM Distillation for One-step Diffusion Models
Sirui Xie, Zhisheng Xiao, Diederik P Kingma +6
While diffusion models can learn complex distributions, sampling requires a computationally expensive iterative process. Existing distillation methods enable efficient sampling, bu…
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation
Diederik P. Kingma, Ruiqi Gao
To achieve the highest perceptual quality, state-of-the-art diffusion models are optimized with objectives that typically look very different from the maximum likelihood and the Ev…
Imagen Video: High Definition Video Generation with Diffusion Models
Jonathan Ho, William Chan, Chitwan Saharia +8
We present Imagen Video, a text-conditional video generation system based on a cascade of video diffusion models. Given a text prompt, Imagen Video generates high definition videos…
On Distillation of Guided Diffusion Models
Chenlin Meng, Robin Rombach, Ruiqi Gao +4
Classifier-free guided diffusion models have recently been shown to be highly effective at high-resolution image generation, and they have been widely used in large-scale diffusion…
Variational Diffusion Models
Diederik P. Kingma, Tim Salimans, Ben Poole +1
Diffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affi…
How to Train Your Energy-Based Models
Yang Song, Diederik P. Kingma
Energy-Based Models (EBMs), also known as non-normalized probabilistic models, specify probability density or mass functions up to an unknown normalizing constant. Unlike most othe…
Learning Energy-Based Models by Diffusion Recovery Likelihood
Ruiqi Gao, Yang Song, Ben Poole +2
While energy-based models (EBMs) exhibit a number of desirable properties, training and sampling on high-dimensional datasets remains challenging. Inspired by recent progress on di…
Score-Based Generative Modeling through Stochastic Differential Equations
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma +3
Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms a complex data distr…
Wave-Tacotron: Spectrogram-free end-to-end text-to-speech synthesis
Ron J. Weiss, RJ Skerry-Ryan, Eric Battenberg +2
We describe a sequence-to-sequence neural network which directly generates speech waveforms from text inputs. The architecture extends the Tacotron model by incorporating a normali…
On Linear Identifiability of Learned Representations
Geoffrey Roeder, Luke Metz, Diederik P. Kingma
Identifiability is a desirable property of a statistical model: it implies that the true model parameters may be estimated to any desired precision, given sufficient computational…
ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA
Ilyes Khemakhem, Ricardo Pio Monti, Diederik P. Kingma +1
We consider the identifiability theory of probabilistic models and establish sufficient conditions under which the representations learned by a very broad family of conditional ene…
Flow Contrastive Estimation of Energy-Based Models
Ruiqi Gao, Erik Nijkamp, Diederik P. Kingma +3
This paper studies a training method to jointly estimate an energy-based model and a flow-based model, in which the two models are iteratively updated based on a shared adversarial…
Variational Autoencoders and Nonlinear ICA: A Unifying Framework
Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti +1
The framework of variational autoencoders allows us to efficiently learn deep latent-variable models, such that the model's marginal distribution over observed variables fits the d…
An Introduction to Variational Autoencoders
Diederik P. Kingma, Max Welling
Variational autoencoders provide a principled framework for learning deep latent-variable models and corresponding inference models. In this work, we provide an introduction to var…
Glow: Generative Flow with Invertible 1x1 Convolutions
Diederik P. Kingma, Prafulla Dhariwal
Flow-based generative models (Dinh et al., 2014) are conceptually attractive due to tractability of the exact log-likelihood, tractability of exact latent-variable inference, and p…
Learning Sparse Neural Networks through Regularization
Christos Louizos, Max Welling, Diederik P. Kingma
We propose a practical method for norm regularization for neural networks: pruning the network during training by encouraging weights to become exactly zero. Such regularizat…
PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
Tim Salimans, Andrej Karpathy, Xi Chen +1
PixelCNNs are a recently proposed class of powerful generative models with tractable likelihood. Here we discuss our implementation of PixelCNNs which we make available at https://…
Variational Lossy Autoencoder
Xi Chen, Diederik P. Kingma, Tim Salimans +5
Representation learning seeks to expose certain aspects of observed data in a learned representation that's amenable to downstream tasks like classification. For instance, a good r…
Improving Variational Inference with Inverse Autoregressive Flow
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz +3
The framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, in…
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Tim Salimans, Diederik P. Kingma
We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparamet…
Variational Dropout and the Local Reparameterization Trick
Diederik P. Kingma, Tim Salimans, Max Welling
We investigate a local reparameterizaton technique for greatly reducing the variance of stochastic gradients for variational Bayesian inference (SGVB) of a posterior over model par…
Note on Equivalence Between Recurrent Neural Network Time Series Models and Variational Bayesian Models
Jascha Sohl-Dickstein, Diederik P. Kingma
We observe that the standard log likelihood training objective for a Recurrent Neural Network (RNN) model of time series data is equivalent to a variational Bayesian training objec…
Adam: A Method for Stochastic Optimization
Diederik P. Kingma, Jimmy Ba
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is str…