Simple, Distributed, and Accelerated Probabilistic Programming
arXiv:1811.02091
Abstract
We describe a simple, low-level approach for embedding probabilistic programming in a deep learning ecosystem. In particular, we distill probabilistic programming down to a single abstraction---the random variable. Our lightweight implementation in TensorFlow enables numerous applications: a model-parallel variational auto-encoder (VAE) with 2nd-generation tensor processing units (TPUv2s); a data-parallel autoregressive model (Image Transformer) with TPUv2s; and multi-GPU No-U-Turn Sampler (NUTS). For both a state-of-the-art VAE on 64x64 ImageNet and Image Transformer on 256x256 CelebA-HQ, our approach achieves an optimal linear speedup from 1 to 256 TPUv2 chips. With NUTS, we see a 100x speedup on GPUs over Stan and 37x over PyMC3.
arXiv commentAppears in Neural Information Processing Systems, 2018. Code available at http://bit.ly/2JpFipt
References in corpus (17)
- Auto-Encoding Variational Bayes
- Neural Architecture Search with Reinforcement Learning
- Pixel Recurrent Neural Networks
- Pyro: Deep Universal Probabilistic Programming
- Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
- Adaptive Computation Time for Recurrent Neural Networks
- Image Transformer
- Edward: A library for probabilistic modeling, inference, and criticism
- Conditional Computation in Neural Networks for faster models
- OptNet: Differentiable Optimization as a Layer in Neural Networks
- Grammar Variational Autoencoder
- Deep Probabilistic Programming
- Operator Variational Inference
- Theory and Experiments on Vector Quantized Autoencoders
- ZhuSuan: A Library for Bayesian Deep Learning
- Design and Implementation of Probabilistic Programming Language Anglican
- Using probabilistic programs as proposals
Cited by in corpus (12)
- Bayesian Layers: A Module for Neural Network Uncertainty
- MultiVerse: Causal Reasoning using Importance Sampling in Probabilistic Programming
- Probabilistic Programming with Programmable Variational Inference
- Learning Proposals for Probabilistic Programs with Inference Combinators
- Efficient Probabilistic Inference in the Quest for Physics Beyond the Standard Model
- Variationally Inferred Sampling Through a Refined Bound for Probabilistic Programs
- D3p -- A Python Package for Differentially-Private Probabilistic Programming
- Simulation-Based Inference for Global Health Decisions
- Meta-Learning an Inference Algorithm for Probabilistic Programs
- Towards Verified Stochastic Variational Inference for Probabilistic Programs
- Deep Classifiers with Label Noise Modeling and Distance Awareness
- Variational State-Space Models for Localisation and Dense 3D Mapping in 6 DoF