Predicting distributions with Linearizing Belief Networks

Abstract

Conditional belief networks introduce stochastic binary variables in neural networks. Contrary to a classical neural network, a belief network can predict more than the expected value of the output $Y$ given the input $X$. It can predict a distribution of outputs $Y$ which is useful when an input can admit multiple outputs whose average is not necessarily a valid answer. Such networks are particularly relevant to inverse problems such as image prediction for denoising, or text to speech. However, traditional sigmoid belief networks are hard to train and are not suited to continuous problems. This work introduces a new family of networks called linearizing belief nets or LBNs. A LBN decomposes into a deep linear network where each linear unit can be turned on or off by non-deterministic binary latent units. It is a universal approximator of real-valued conditional distributions and can be trained using gradient descent. Moreover, the linear pathways efficiently propagate continuous information and they act as multiplicative skip-connections that help optimization by removing gradient diffusion. This yields a model which trains efficiently and improves the state-of-the-art on image denoising and facial expression generation with the Toronto faces dataset.

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Deep Generative Stochastic Networks Trainable by Backprop
• Generalized Denoising Auto-Encoders as Generative Models
• Techniques for Learning Binary Stochastic Feedforward Neural Networks

Cited by in corpus (5)

• Language Modeling with Gated Convolutional Networks
• Improving Variational Auto-Encoders using Householder Flow
• Improving Variational Auto-Encoders using convex combination linear Inverse Autoregressive Flow
• A Deeper Look at Salient Object Detection: Bi-stream Network with a Small Training Dataset
• Left ventricle segmentation By modelling uncertainty in prediction of deep convolutional neural networks and adaptive thresholding inference