Conditional Restricted Boltzmann Machines for Structured Output Prediction

Abstract

Conditional Restricted Boltzmann Machines (CRBMs) are rich probabilistic models that have recently been applied to a wide range of problems, including collaborative filtering, classification, and modeling motion capture data. While much progress has been made in training non-conditional RBMs, these algorithms are not applicable to conditional models and there has been almost no work on training and generating predictions from conditional RBMs for structured output problems. We first argue that standard Contrastive Divergence-based learning may not be suitable for training CRBMs. We then identify two distinct types of structured output prediction problems and propose an improved learning algorithm for each. The first problem type is one where the output space has arbitrary structure but the set of likely output configurations is relatively small, such as in multi-label classification. The second problem is one where the output space is arbitrarily structured but where the output space variability is much greater, such as in image denoising or pixel labeling. We show that the new learning algorithms can work much better than Contrastive Divergence on both types of problems.

References in corpus (2)

• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Autotagging music with conditional restricted Boltzmann machines

Cited by in corpus (13)

• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Deep learning for comprehensive forecasting of Alzheimer's Disease progression
• Deep Learning of Representations: Looking Forward
• Deep Learning for Semantic Part Segmentation with High-Level Guidance
• Improving Maritime Traffic Emission Estimations on Missing Data with CRBMs
• Energy-based Models for Video Anomaly Detection
• Modeling Disease Progression in Mild Cognitive Impairment and Alzheimer's Disease with Digital Twins
• Belief Propagation in Conditional RBMs for Structured Prediction
• Probabilistic Discriminative Learning with Layered Graphical Models
• Deep Neural Networks Regularization for Structured Output Prediction
• Sequential Labeling with online Deep Learning
• A Generative Model for Multi-Dialect Representation
• Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search