Learning Feature Hierarchies with Centered Deep Boltzmann Machines

Abstract

Deep Boltzmann machines are in principle powerful models for extracting the hierarchical structure of data. Unfortunately, attempts to train layers jointly (without greedy layer-wise pretraining) have been largely unsuccessful. We propose a modification of the learning algorithm that initially recenters the output of the activation functions to zero. This modification leads to a better conditioned Hessian and thus makes learning easier. We test the algorithm on real data and demonstrate that our suggestion, the centered deep Boltzmann machine, learns a hierarchy of increasingly abstract representations and a better generative model of data.

References in corpus (3)

• Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
• Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent
• Annealed Importance Sampling

Cited by in corpus (18)

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
• Deep Generative Stochastic Networks Trainable by Backprop
• Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
• Application of Quantum Annealing to Training of Deep Neural Networks
• Deep Learning of Representations: Looking Forward
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Metric-Free Natural Gradient for Joint-Training of Boltzmann Machines
• Neural Autoregressive Distribution Estimation
• Wasserstein Training of Boltzmann Machines
• Prediction of breast cancer recurrence using Classification Restricted Boltzmann Machine with Dropping
• Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians
• A Coordinate-Free Construction of Scalable Natural Gradient
• Restricted Boltzmann Machine and Deep Belief Network: Tutorial and Survey
• GSNs : Generative Stochastic Networks
• A Hippocampus Model for Online One-Shot Storage of Pattern Sequences
• Deep Learning Moment Closure Approximations using Dynamic Boltzmann Distributions
• Modeling correlations in spontaneous activity of visual cortex with centered Gaussian-binary deep Boltzmann machines