Better Mixing via Deep Representations

Abstract

It has previously been hypothesized, and supported with some experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce faster-mixing Markov chains. Consequently, mixing would be more efficient at higher levels of representation. To better understand why and how this is happening, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing and interpolating between samples.

References in corpus (1)

• A Generative Process for Sampling Contractive Auto-Encoders

Cited by in corpus (84)

• Conditional Generative Adversarial Nets
• Generative Adversarial Text to Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• NICE: Non-linear Independent Components Estimation
• Guiding Deep Learning System Testing using Surprise Adequacy
• A note on the evaluation of generative models
• Generative Moment Matching Networks
• Detecting Adversarial Samples from Artifacts
• Deep Generative Stochastic Networks Trainable by Backprop
• Adversarial Autoencoders
• Opportunities and challenges for quantum-assisted machine learning in near-term quantum computers
• How to Construct Deep Recurrent Neural Networks
• Generalized Denoising Auto-Encoders as Generative Models
• Demystifying MMD GANs
• Discovering Hidden Factors of Variation in Deep Networks
• How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
• Deep Learning of Representations: Looking Forward
• Representation Learning: A Review and New Perspectives
• What Regularized Auto-Encoders Learn from the Data Generating Distribution
• Deep Directed Generative Models with Energy-Based Probability Estimation
• BubGAN: Bubble Generative Adversarial Networks for Synthesizing Realistic Bubbly Flow Images
• Variational Auto-encoded Deep Gaussian Processes
• Maximum Entropy Generators for Energy-Based Models
• Learning Implicit Fields for Generative Shape Modeling
• Implicit Maximum Likelihood Estimation
• Reliable Fidelity and Diversity Metrics for Generative Models
• Adversarial Examples in Modern Machine Learning: A Review
• GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data
• Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling
• Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model
• Deep Learning Convolutional Networks for Multiphoton Microscopy Vasculature Segmentation
• Tackling Over-pruning in Variational Autoencoders
• Accelerated physical emulation of Bayesian inference in spiking neural networks
• MGGAN: Solving Mode Collapse using Manifold Guided Training
• Fortified Networks: Improving the Robustness of Deep Networks by Modeling the Manifold of Hidden Representations
• Learning Neural Generative Dynamics for Molecular Conformation Generation
• Improved Training of Generative Adversarial Networks Using Representative Features
• Reweighted Wake-Sleep
• ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes
• Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation
• Task dependent Deep LDA pruning of neural networks
• Regularizing Deep Networks with Semantic Data Augmentation
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models
• Iterative Neural Autoregressive Distribution Estimator (NADE-k)
• SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editing
• Benchmarking Regression Methods: A comparison with CGAN
• Deep Directed Generative Autoencoders
• Bounding the Test Log-Likelihood of Generative Models
• Generalization in Machine Learning via Analytical Learning Theory
• Why Regularized Auto-Encoders learn Sparse Representation?
• DeLiGAN : Generative Adversarial Networks for Diverse and Limited Data
• Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net
• Joint Contrastive Learning with Infinite Possibilities
• Categorical Perception: A Groundwork for Deep Learning
• Optimal transport maps for distribution preserving operations on latent spaces of Generative Models
• A Brief Introduction to Generative Models
• Multimodal Transitions for Generative Stochastic Networks
• NeRD: Neural Representation of Distribution for Medical Image Segmentation
• CFSNet: Toward a Controllable Feature Space for Image Restoration
• DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Sampling
• Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning
• Representation learning for improved interpretability and classification accuracy of clinical factors from EEG
• Learning Latent Space Energy-Based Prior Model
• Classify and Generate: Using Classification Latent Space Representations for Image Generations
• Generative Mixture of Networks
• Exploiting GAN Internal Capacity for High-Quality Reconstruction of Natural Images
• Channel-Recurrent Autoencoding for Image Modeling
• GSNs : Generative Stochastic Networks
• Regression via Implicit Models and Optimal Transport Cost Minimization
• Unsupervised Pretraining Encourages Moderate-Sparseness
• Neural Machine Translation: A Review and Survey
• Shape-aware Generative Adversarial Networks for Attribute Transfer
• Improving Calibration and Out-of-Distribution Detection in Medical Image Segmentation with Convolutional Neural Networks
• PassFlow: Guessing Passwords with Generative Flows
• Out-distribution training confers robustness to deep neural networks
• Neural Crossbreed: Neural Based Image Metamorphosis
• Grow-Push-Prune: aligning deep discriminants for effective structural network compression
• Multi-class Novelty Detection Using Mix-up Technique
• Open Set Domain Adaptation for Image and Action Recognition
• GFlowNet Foundations
• Concept Formation and Dynamics of Repeated Inference in Deep Generative Models
• Adiabatic Persistent Contrastive Divergence Learning
• Deep Adaptive Inference Networks for Single Image Super-Resolution