Composing graphical models with neural networks for structured representations and fast inference

Abstract

We propose a general modeling and inference framework that composes probabilistic graphical models with deep learning methods and combines their respective strengths. Our model family augments graphical structure in latent variables with neural network observation models. For inference, we extend variational autoencoders to use graphical model approximating distributions with recognition networks that output conjugate potentials. All components of these models are learned simultaneously with a single objective, giving a scalable algorithm that leverages stochastic variational inference, natural gradients, graphical model message passing, and the reparameterization trick. We illustrate this framework with several example models and an application to mouse behavioral phenotyping.

v5 fixes tex compilation bugs and also a math bug in the statement and proof of Prop. 4.1 (and D.3). v4 adds two paragraphs to the related work section and fixes typos in the appendices. v3 fixes some typos in the appendices. v2 is a rewrite from v1 to be more readable and to include detailed appendices

References in corpus (1)

• DRAW: A Recurrent Neural Network For Image Generation

Cited by in corpus (89)

• An Introduction to Variational Autoencoders
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• Recent Advances in Autoencoder-Based Representation Learning
• Harnessing Deep Neural Networks with Logic Rules
• MR image reconstruction using deep density priors
• Dynamical Variational Autoencoders: A Comprehensive Review
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• Towards a Neural Statistician
• OptNet: Differentiable Optimization as a Layer in Neural Networks
• A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
• Foraging as an evidence accumulation process
• Symbol Emergence in Cognitive Developmental Systems: a Survey
• Information Constraints on Auto-Encoding Variational Bayes
• Deep Structural Causal Models for Tractable Counterfactual Inference
• Learning Plannable Representations with Causal InfoGAN
• Time2Graph: Revisiting Time Series Modeling with Dynamic Shapelets
• Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
• Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Disentangling Disentanglement in Variational Autoencoders
• Disentanglement by Nonlinear ICA with General Incompressible-flow Networks (GIN)
• Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
• Learning identifiable and interpretable latent models of high-dimensional neural activity using pi-VAE
• Variational Sequential Monte Carlo
• Practical Lossless Compression with Latent Variables using Bits Back Coding
• Linear dynamical neural population models through nonlinear embeddings
• Robust Training of Vector Quantized Bottleneck Models
• Discrete Variational Autoencoders
• Nonparametric Variational Auto-encoders for Hierarchical Representation Learning
• COT-GAN: Generating Sequential Data via Causal Optimal Transport
• ZhuSuan: A Library for Bayesian Deep Learning
• Semi-Amortized Variational Autoencoders
• A Tutorial on Deep Latent Variable Models of Natural Language
• Pixel Deconvolutional Networks
• Nimble: Efficiently Compiling Dynamic Neural Networks for Model Inference
• Probabilistic Model-Agnostic Meta-Learning
• Disentangling Identifiable Features from Noisy Data with Structured Nonlinear ICA
• Prediction and Control with Temporal Segment Models
• Scalable Gaussian Process Variational Autoencoders
• Unsupervised Word Segmentation from Speech with Attention
• Behavior Priors for Efficient Reinforcement Learning
• Conditional Sum-Product Networks: Imposing Structure on Deep Probabilistic Architectures
• Monotone operator equilibrium networks
• Recurrent Neural Processes
• Graphical Normalizing Flows
• Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling
• Adaptive Path-Integral Autoencoder: Representation Learning and Planning for Dynamical Systems
• CompILE: Compositional Imitation Learning and Execution
• Deep Probabilistic Logic: A Unifying Framework for Indirect Supervision
• Nonlinear Evolution via Spatially-Dependent Linear Dynamics for Electrophysiology and Calcium Data
• Deep Quantization: Encoding Convolutional Activations with Deep Generative Model
• Inducing Interpretable Representations with Variational Autoencoders
• Gaussian mixture models with Wasserstein distance
• Semi-crowdsourced Clustering with Deep Generative Models
• The Differentiable Cross-Entropy Method
• LeMoNADe: Learned Motif and Neuronal Assembly Detection in calcium imaging videos
• Scaling Factorial Hidden Markov Models: Stochastic Variational Inference without Messages
• Implicitly Defined Layers in Neural Networks
• Unsupervised Recurrent Neural Network Grammars
• Direct Optimization through $\arg \max$ for Discrete Variational Auto-Encoder
• Deep Random Splines for Point Process Intensity Estimation of Neural Population Data
• The LORACs prior for VAEs: Letting the Trees Speak for the Data
• Streaming Adaptive Nonparametric Variational Autoencoder
• Variational online learning of neural dynamics
• Continuous Graph Flow
• Feedback from Pixels: Output Regulation via Learning-Based Scene View Synthesis
• Differentially Private Variational Autoencoders with Term-wise Gradient Aggregation
• Stochastic Sequential Neural Networks with Structured Inference
• Learning interaction rules from multi-animal trajectories via augmented behavioral models
• Hidden Markov Neural Networks
• Bidirectional Inference Networks: A Class of Deep Bayesian Networks for Health Profiling
• Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax
• Flexible mean field variational inference using mixtures of non-overlapping exponential families
• Learning Calibratable Policies using Programmatic Style-Consistency
• Adversarially-learned Inference via an Ensemble of Discrete Undirected Graphical Models
• Region-based Energy Neural Network for Approximate Inference
• Unsupervised Learning of Neurosymbolic Encoders
• LaDDer: Latent Data Distribution Modelling with a Generative Prior
• Amortized backward variational inference in nonlinear state-space models
• A unified algorithm framework for quality control of sensor data for behavioural clinimetric testing
• Scalable Explanation of Inferences on Large Graphs
• Online Unsupervised Learning of Visual Representations and Categories
• Dynamic Variational Autoencoders for Visual Process Modeling
• Optical Mouse: 3D Mouse Pose From Single-View Video
• Self-supervised self-supervision by combining deep learning and probabilistic logic
• Learning Correlated Latent Representations with Adaptive Priors
• From Demonstrations to Task-Space Specifications: Using Causal Analysis to Extract Rule Parameterization from Demonstrations
• Investigating naturalistic hand movements by behavior mining in long-term video and neural recordings
• A Probabilistic Model of Cardiac Physiology and Electrocardiograms