Hidden Markov Neural Networks

Abstract

We define an evolving in-time Bayesian neural network called a Hidden Markov Neural Network, which addresses the crucial challenge in time-series forecasting and continual learning: striking a balance between adapting to new data and appropriately forgetting outdated information. This is achieved by modelling the weights of a neural network as the hidden states of a Hidden Markov model, with the observed process defined by the available data. A filtering algorithm is employed to learn a variational approximation of the evolving-in-time posterior distribution over the weights. By leveraging a sequential variant of Bayes by Backprop, enriched with a stronger regularization technique called variational DropConnect, Hidden Markov Neural Networks achieve robust regularization and scalable inference. Experiments on MNIST, dynamic classification tasks, and next-frame forecasting in videos demonstrate that Hidden Markov Neural Networks provide strong predictive performance while enabling effective uncertainty quantification.

References in corpus (16)

• Scikit-learn: Machine Learning in Python
• Auto-Encoding Variational Bayes
• Overcoming catastrophic forgetting in neural networks
• Improving neural networks by preventing co-adaptation of feature detectors
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Weight Uncertainty in Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Variational Continual Learning
• Composing graphical models with neural networks for structured representations and fast inference
• Can local particle filters beat the curse of dimensionality?
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• A Bayesian Federated Learning Framework with Online Laplace Approximation
• Convergence Rates of Variational Inference in Sparse Deep Learning
• DropConnect Is Effective in Modeling Uncertainty of Bayesian Deep Networks
• Exploiting locality in high-dimensional factorial hidden Markov models