Tighter Variational Bounds are Not Necessarily Better

Abstract

We provide theoretical and empirical evidence that using tighter evidence lower bounds (ELBOs) can be detrimental to the process of learning an inference network by reducing the signal-to-noise ratio of the gradient estimator. Our results call into question common implicit assumptions that tighter ELBOs are better variational objectives for simultaneous model learning and inference amortization schemes. Based on our insights, we introduce three new algorithms: the partially importance weighted auto-encoder (PIWAE), the multiply importance weighted auto-encoder (MIWAE), and the combination importance weighted auto-encoder (CIWAE), each of which includes the standard importance weighted auto-encoder (IWAE) as a special case. We show that each can deliver improvements over IWAE, even when performance is measured by the IWAE target itself. Furthermore, our results suggest that PIWAE may be able to deliver simultaneous improvements in the training of both the inference and generative networks.

arXiv commentTo appear at ICML 2018

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Auxiliary Deep Generative Models
• Variational Lossy Autoencoder
• Improving Variational Inference with Inverse Autoregressive Flow
• Towards Conceptual Compression
• Auto-Encoding Sequential Monte Carlo
• Rényi Divergence Variational Inference
• Inference Networks for Sequential Monte Carlo in Graphical Models
• On Nesting Monte Carlo Estimators
• Filtering Variational Objectives

Cited by in corpus (13)

• A Deep Learning Algorithm for High-Dimensional Exploratory Item Factor Analysis
• A Tutorial on Deep Latent Variable Models of Natural Language
• Amortized Variational Inference: A Systematic Review
• Semi-Implicit Variational Inference
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling
• Differentiable Particle Filtering without Modifying the Forward Pass
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Failure Modes of Variational Autoencoders and Their Effects on Downstream Tasks
• Amortised Learning by Wake-Sleep
• PAC${}^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime
• Neighbor Embedding Variational Autoencoder
• Mutual Information Constraints for Monte-Carlo Objectives