Stochastic Neural Network with Kronecker Flow
arXiv:1906.04282
Abstract
Recent advances in variational inference enable the modelling of highly structured joint distributions, but are limited in their capacity to scale to the high-dimensional setting of stochastic neural networks. This limitation motivates a need for scalable parameterizations of the noise generation process, in a manner that adequately captures the dependencies among the various parameters. In this work, we address this need and present the Kronecker Flow, a generalization of the Kronecker product to invertible mappings designed for stochastic neural networks. We apply our method to variational Bayesian neural networks on predictive tasks, PAC-Bayes generalization bound estimation, and approximate Thompson sampling in contextual bandits. In all setups, our methods prove to be competitive with existing methods and better than the baselines.
arXiv commentProceedings of the 23rdInternational Conference on ArtificialIntelligence and Statistics (AISTATS) 2020
References in corpus (19)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Conditional Image Generation with PixelCNN Decoders
- Variational Inference with Normalizing Flows
- Density estimation using Real NVP
- Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
- VIME: Variational Information Maximizing Exploration
- Variational Dropout and the Local Reparameterization Trick
- (More) Efficient Reinforcement Learning via Posterior Sampling
- Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
- Invertible Residual Networks
- Functional Variational Bayesian Neural Networks
- Sylvester Normalizing Flows for Variational Inference
- Stronger generalization bounds for deep nets via a compression approach
- Implicit Weight Uncertainty in Neural Networks
- Block Neural Autoregressive Flow
- Learning Gaussian Processes by Minimizing PAC-Bayesian Generalization Bounds
- PAC-Bayes with Backprop
- Bayesian Hypernetworks
- Randomized Value Functions via Multiplicative Normalizing Flows