Stochastic Sequential Neural Networks with Structured Inference
arXiv:1705.08695
Abstract
Unsupervised structure learning in high-dimensional time series data has attracted a lot of research interests. For example, segmenting and labelling high dimensional time series can be helpful in behavior understanding and medical diagnosis. Recent advances in generative sequential modeling have suggested to combine recurrent neural networks with state space models (e.g., Hidden Markov Models). This combination can model not only the long term dependency in sequential data, but also the uncertainty included in the hidden states. Inheriting these advantages of stochastic neural sequential models, we propose a structured and stochastic sequential neural network, which models both the long-term dependencies via recurrent neural networks and the uncertainty in the segmentation and labels via discrete random variables. For accurate and efficient inference, we present a bi-directional inference network by reparamterizing the categorical segmentation and labels with the recent proposed Gumbel-Softmax approximation and resort to the Stochastic Gradient Variational Bayes. We evaluate the proposed model in a number of tasks, including speech modeling, automatic segmentation and labeling in behavior understanding, and sequential multi-objects recognition. Experimental results have demonstrated that our proposed model can achieve significant improvement over the state-of-the-art methods.
References in corpus (22)
- Adam: A Method for Stochastic Optimization
- Categorical Reparameterization with Gumbel-Softmax
- DRAW: A Recurrent Neural Network For Image Generation
- A Recurrent Latent Variable Model for Sequential Data
- Hierarchical Multiscale Recurrent Neural Networks
- Bayesian Nonparametric Inference of Switching Linear Dynamical Systems
- Composing graphical models with neural networks for structured representations and fast inference
- Learning Stochastic Recurrent Networks
- Variational Recurrent Auto-Encoders
- Sequential Neural Models with Stochastic Layers
- Variational inference for Monte Carlo objectives
- Deep Kalman Filters
- Blocks and Fuel: Frameworks for deep learning
- Black box variational inference for state space models
- Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data
- Inference in Hidden Markov Models with Explicit State Duration Distributions
- Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models
- Neural Adaptive Sequential Monte Carlo
- Discrete Variational Autoencoders
- MuProp: Unbiased Backpropagation for Stochastic Neural Networks
- Stochastic Generative Hashing
- Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions