Learning Wake-Sleep Recurrent Attention Models
arXiv:1509.06812
Abstract
Despite their success, convolutional neural networks are computationally expensive because they must examine all image locations. Stochastic attention-based models have been shown to improve computational efficiency at test time, but they remain difficult to train because of intractable posterior inference and high variance in the stochastic gradient estimates. Borrowing techniques from the literature on training deep generative models, we present the Wake-Sleep Recurrent Attention Model, a method for training stochastic attention networks which improves posterior inference and which reduces the variability in the stochastic gradients. We show that our method can greatly speed up the training time for stochastic attention networks in the domains of image classification and caption generation.
arXiv commentTo appear in NIPS 2015
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Recurrent Models of Visual Attention
- DRAW: A Recurrent Neural Network For Image Generation
- Importance Weighted Autoencoders
- Neural Variational Inference and Learning in Belief Networks
- The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
- Variational Bayesian Inference with Stochastic Search
- Learning Generative Models with Visual Attention
- Reweighted Wake-Sleep