Using stochastic computation graphs formalism for optimization of sequence-to-sequence model
arXiv:1711.07724
Abstract
Variety of machine learning problems can be formulated as an optimization task for some (surrogate) loss function. Calculation of loss function can be viewed in terms of stochastic computation graphs (SCG). We use this formalism to analyze a problem of optimization of famous sequence-to-sequence model with attention and propose reformulation of the task. Examples are given for machine translation (MT). Our work provides a unified view on different optimization approaches for sequence-to-sequence models and could help researchers in developing new network architectures with embedded stochastic nodes.
arXiv commentPresented at 10th NIPS Workshop on Optimization for Machine Learning (NIPS 2017)
References in corpus (5)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- A Deep Reinforced Model for Abstractive Summarization
- Enriching Word Vectors with Subword Information
- REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models