A Theory of Generative ConvNet

Abstract

We show that a generative random field model, which we call generative ConvNet, can be derived from the commonly used discriminative ConvNet, by assuming a ConvNet for multi-category classification and assuming one of the categories is a base category generated by a reference distribution. If we further assume that the non-linearity in the ConvNet is Rectified Linear Unit (ReLU) and the reference distribution is Gaussian white noise, then we obtain a generative ConvNet model that is unique among energy-based models: The model is piecewise Gaussian, and the means of the Gaussian pieces are defined by an auto-encoder, where the filters in the bottom-up encoding become the basis functions in the top-down decoding, and the binary activation variables detected by the filters in the bottom-up convolution process become the coefficients of the basis functions in the top-down deconvolution process. The Langevin dynamics for sampling the generative ConvNet is driven by the reconstruction error of this auto-encoder. The contrastive divergence learning of the generative ConvNet reconstructs the training images by the auto-encoder. The maximum likelihood learning algorithm can synthesize realistic natural image patterns.

Cited by in corpus (71)

• Denoising Diffusion Probabilistic Models
• Energy-based Out-of-distribution Detection
• Implicit Generation and Generalization in Energy-Based Models
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• How to Train Your Energy-Based Models
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training
• Can multi-label classification networks know what they don't know?
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• Learning Neural Generative Dynamics for Molecular Conformation Generation
• Improved Contrastive Divergence Training of Energy Based Models
• Hybrid Discriminative-Generative Training via Contrastive Learning
• Meta-Learning Deep Energy-Based Memory Models
• Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• Residual Energy-Based Models for Text
• Learning Energy-Based Models by Diffusion Recovery Likelihood
• Inferring effective couplings with Restricted Boltzmann Machines
• How to Train Your Energy-Based Model for Regression
• Towards Interpretable R-CNN by Unfolding Latent Structures
• Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learning
• Unsupervised Learning of Compositional Energy Concepts
• Dense Uncertainty Estimation
• Synthesizing Dynamic Patterns by Spatial-Temporal Generative ConvNet
• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• Stochastic Security: Adversarial Defense Using Long-Run Dynamics of Energy-Based Models
• CAGAN: Text-To-Image Generation with Combined Attention GANs
• EBM-Fold: Fully-Differentiable Protein Folding Powered by Energy-based Models
• Compositional Visual Generation and Inference with Energy Based Models
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Energy-Based Models for Continual Learning
• Convolutional Bipartite Attractor Networks
• Trajectory Prediction with Latent Belief Energy-Based Model
• Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning
• Flow Contrastive Estimation of Energy-Based Models
• Learning Latent Space Energy-Based Prior Model
• Unpaired Image-to-Image Translation via Latent Energy Transport
• Semi-supervised Learning by Latent Space Energy-Based Model of Symbol-Vector Coupling
• Efficient Learning of Generative Models via Finite-Difference Score Matching
• Energy-Based Models with Applications to Speech and Language Processing
• Dual Contradistinctive Generative Autoencoder
• Energy-constrained Self-training for Unsupervised Domain Adaptation
• Divergence Triangle for Joint Training of Generator Model, Energy-based Model, and Inference Model
• Motion-Based Generator Model: Unsupervised Disentanglement of Appearance, Trackable and Intrackable Motions in Dynamic Patterns
• Contrastive Divergence Learning is a Time Reversal Adversarial Game
• Learning Cycle-Consistent Cooperative Networks via Alternating MCMC Teaching for Unsupervised Cross-Domain Translation
• Conditional Generative ConvNets for Exemplar-based Texture Synthesis
• Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Execution
• Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• Introspective Generative Modeling: Decide Discriminatively
• Energy-Based Processes for Exchangeable Data
• Wasserstein Introspective Neural Networks
• Explaining the effects of non-convergent sampling in the training of Energy-Based Models
• Learning Energy-Based Models as Generative ConvNets via Multi-grid Modeling and Sampling
• Bridging Explicit and Implicit Deep Generative Models via Neural Stein Estimators
• Learning Energy-based Spatial-Temporal Generative ConvNets for Dynamic Patterns
• A Tale of Three Probabilistic Families: Discriminative, Descriptive and Generative Models
• Generative Modeling by Inclusive Neural Random Fields with Applications in Image Generation and Anomaly Detection
• Estimation of Non-Normalized Mixture Models and Clustering Using Deep Representation
• Autoencoding Under Normalization Constraints
• An Energy-Based View of Graph Neural Networks
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
• Set Prediction without Imposing Structure as Conditional Density Estimation
• Learning Proposals for Practical Energy-Based Regression
• EBMs Trained with Maximum Likelihood are Generator Models Trained with a Self-adverserial Loss
• An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training
• Generative Max-Mahalanobis Classifiers for Image Classification, Generation and More
• RNA Alternative Splicing Prediction with Discrete Compositional Energy Network
• Representation Learning: A Statistical Perspective
• FRAME Revisited: An Interpretation View Based on Particle Evolution