Techniques for Learning Binary Stochastic Feedforward Neural Networks

Abstract

Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output is important. (3) Stochasticity has been shown to be an excellent regularizer, which makes generalization performance potentially better in general. However, training stochastic networks is considerably more difficult. We study training using M samples of hidden activations per input. We show that the case M=1 leads to a fundamentally different behavior where the network tries to avoid stochasticity. We propose two new estimators for the training gradient and propose benchmark tests for comparing training algorithms. Our experiments confirm that training stochastic networks is difficult and show that the proposed two estimators perform favorably among all the five known estimators.

References in corpus (5)

• Improving neural networks by preventing co-adaptation of feature detectors
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• Estimating or Propagating Gradients Through Stochastic Neurons

Cited by in corpus (35)

• Semi-Supervised Learning with Ladder Networks
• Learning for Video Compression
• Variational inference for Monte Carlo objectives
• Variable Rate Image Compression with Recurrent Neural Networks
• Robust Convolutional Neural Networks under Adversarial Noise
• A Hybrid Convolutional Variational Autoencoder for Text Generation
• Discrete Variational Autoencoders
• MuProp: Unbiased Backpropagation for Stochastic Neural Networks
• Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objective
• Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning
• Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series
• A RAD approach to deep mixture models
• Image compression optimized for 3D reconstruction by utilizing deep neural networks
• PixelVAE++: Improved PixelVAE with Discrete Prior
• Stochastic Generative Hashing
• Generating Natural Language Adversarial Examples on a Large Scale with Generative Models
• BINet: a binary inpainting network for deep patch-based image compression
• Modelling Latent Translations for Cross-Lingual Transfer
• Layer-wise Learning of Stochastic Neural Networks with Information Bottleneck
• Predicting distributions with Linearizing Belief Networks
• Learning Deep Generative Models with Annealed Importance Sampling
• Exploring the Back Alleys: Analysing The Robustness of Alternative Neural Network Architectures against Adversarial Attacks
• Gibbs Sampling the Posterior of Neural Networks
• Improved Gradient-Based Optimization Over Discrete Distributions
• Simplified Stochastic Feedforward Neural Networks
• No Multiplication? No Floating Point? No Problem! Training Networks for Efficient Inference
• Select and Attend: Towards Controllable Content Selection in Text Generation
• Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks
• Variational Rejection Sampling
• Iterative Refinement of the Approximate Posterior for Directed Belief Networks
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax
• Fitting summary statistics of neural data with a differentiable spiking network simulator
• Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts
• A Fourier View of REINFORCE