Neural Machine Translation and Sequence-to-sequence Models: A Tutorial

Abstract

This tutorial introduces a new and powerful set of techniques variously called "neural machine translation" or "neural sequence-to-sequence models". These techniques have been used in a number of tasks regarding the handling of human language, and can be a powerful tool in the toolbox of anyone who wants to model sequential data of some sort. The tutorial assumes that the reader knows the basics of math and programming, but does not assume any particular experience with neural networks or natural language processing. It attempts to explain the intuition behind the various methods covered, then delves into them with enough mathematical detail to understand them concretely, and culiminates with a suggestion for an implementation exercise, where readers can test that they understood the content in practice.

65 Pages

References in corpus (21)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• LSTM: A Search Space Odyssey
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• An overview of gradient descent optimization algorithms
• Neural Architecture Search with Reinforcement Learning
• From Frequency to Meaning: Vector Space Models of Semantics
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• A Neural Conversational Model
• DRAW: A Recurrent Neural Network For Image Generation
• Visualizing and Understanding Recurrent Networks
• Transition-Based Dependency Parsing with Stack Long Short-Term Memory
• Semi-supervised Sequence Learning
• DyNet: The Dynamic Neural Network Toolkit
• Neural Machine Translation in Linear Time
• Toward Multilingual Neural Machine Translation with Universal Encoder and Decoder
• When Are Tree Structures Necessary for Deep Learning of Representations?
• Reward Augmented Maximum Likelihood for Neural Structured Prediction
• Classes for Fast Maximum Entropy Training

Cited by in corpus (23)

• Deep Learning-based Vehicle Behaviour Prediction For Autonomous Driving Applications: A Review
• A Survey of Domain Adaptation for Neural Machine Translation
• Deep Reinforcement Learning for Electric Vehicle Routing Problem with Time Windows
• Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
• Transformer-based Map Matching Model with Limited Ground-Truth Data using Transfer-Learning Approach
• Learning to Generate Corrective Patches using Neural Machine Translation
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• Evaluating Text GANs as Language Models
• A Comprehensive Survey of Multilingual Neural Machine Translation
• On Compositionality in Neural Machine Translation
• PharmMT: A Neural Machine Translation Approach to Simplify Prescription Directions
• On the Dimensionality of Embeddings for Sparse Features and Data
• First Experiments with Neural Translation of Informal to Formal Mathematics
• Optimized Broadcast for Deep Learning Workloads on Dense-GPU InfiniBand Clusters: MPI or NCCL?
• Towards Neural Decompilation
• A Brief Survey of Multilingual Neural Machine Translation
• Attention Forcing for Sequence-to-sequence Model Training
• Learning Vehicle Routing Problems using Policy Optimisation
• Neural Machine Translation: A Review and Survey
• Attention Forcing for Machine Translation
• Multi-Issue Bargaining With Deep Reinforcement Learning
• Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time Series Forecasting
• Incorporating Chinese Radicals Into Neural Machine Translation: Deeper Than Character Level