On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Abstract

Neural machine translation is a relatively new approach to statistical machine translation based purely on neural networks. The neural machine translation models often consist of an encoder and a decoder. The encoder extracts a fixed-length representation from a variable-length input sentence, and the decoder generates a correct translation from this representation. In this paper, we focus on analyzing the properties of the neural machine translation using two models; RNN Encoder--Decoder and a newly proposed gated recursive convolutional neural network. We show that the neural machine translation performs relatively well on short sentences without unknown words, but its performance degrades rapidly as the length of the sentence and the number of unknown words increase. Furthermore, we find that the proposed gated recursive convolutional network learns a grammatical structure of a sentence automatically.

arXiv commentEighth Workshop on Syntax, Semantics and Structure in Statistical Translation (SSST-8)

References in corpus (1)

• Sequence Transduction with Recurrent Neural Networks

Cited by in corpus (281)

• Neural Machine Translation by Jointly Learning to Align and Translate
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• The Curious Case of Neural Text Degeneration
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• Recurrent Neural Networks with Top-k Gains for Session-based Recommendations
• Comparative Study of CNN and RNN for Natural Language Processing
• VulDeePecker: A Deep Learning-Based System for Vulnerability Detection
• Skip-Thought Vectors
• Deep Learning in Video Multi-Object Tracking: A Survey
• DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset
• Light Gated Recurrent Units for Speech Recognition
• A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction
• A Comprehensive Survey of Scene Graphs: Generation and Application
• Machine Learning for Wireless Communications in the Internet of Things: A Comprehensive Survey
• Working Memory Connections for LSTM
• Graph networks as learnable physics engines for inference and control
• Depthwise Separable Convolutions for Neural Machine Translation
• SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction
• Learnable pooling with Context Gating for video classification
• Multimodal Residual Learning for Visual QA
• Unitary Evolution Recurrent Neural Networks
• End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
• Learning What and Where to Draw
• Multi-Task Cross-Lingual Sequence Tagging from Scratch
• Sequence to Sequence -- Video to Text
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Maximum-Likelihood Augmented Discrete Generative Adversarial Networks
• Deep Complex Networks
• Fast Domain Adaptation for Neural Machine Translation
• Social Interactions for Autonomous Driving: A Review and Perspectives
• Analyzing Input and Output Representations for Speech-Driven Gesture Generation
• Full-Capacity Unitary Recurrent Neural Networks
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• Machine learning based hyperspectral image analysis: A survey
• Gait Recognition Based on Deep Learning: A Survey
• Improving Distantly Supervised Relation Extraction using Word and Entity Based Attention
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Exploring Interpretable LSTM Neural Networks over Multi-Variable Data
• Recurrent Dropout without Memory Loss
• Trellis Networks for Sequence Modeling
• Compositional Vector Space Models for Knowledge Base Completion
• Robust Scene Text Recognition with Automatic Rectification
• Optimizing Performance of Recurrent Neural Networks on GPUs
• Deep Learning for Sequential Recommendation: Algorithms, Influential Factors, and Evaluations
• Multilingual Neural Machine Translation with Task-Specific Attention
• Deep Neural Approaches to Relation Triplets Extraction: A Comprehensive Survey
• You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion
• Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks
• Deep Learning for Environmentally Robust Speech Recognition: An Overview of Recent Developments
• ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA
• Multilingual Extractive Reading Comprehension by Runtime Machine Translation
• Convolutional Recurrent Neural Networks for Music Classification
• Tensor-Train Recurrent Neural Networks for Video Classification
• Challenges in Data-to-Document Generation
• Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• Imposing higher-level Structure in Polyphonic Music Generation using Convolutional Restricted Boltzmann Machines and Constraints
• Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothness
• Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• Larger-Context Language Modelling
• DNA-Level Splice Junction Prediction using Deep Recurrent Neural Networks
• Embedding Word Similarity with Neural Machine Translation
• Reconstruction Network for Video Captioning
• Exploring Question Understanding and Adaptation in Neural-Network-Based Question Answering
• RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model
• DeepSpline: Data-Driven Reconstruction of Parametric Curves and Surfaces
• Modeling Time Series Similarity with Siamese Recurrent Networks
• Gibbs-Helmholtz Graph Neural Network: capturing the temperature dependency of activity coefficients at infinite dilution
• Incorporating Discrete Translation Lexicons into Neural Machine Translation
• Coverage Embedding Models for Neural Machine Translation
• Linguistic Knowledge as Memory for Recurrent Neural Networks
• Learning Long Term Dependencies via Fourier Recurrent Units
• Learning Deep Neural Networks for Vehicle Re-ID with Visual-spatio-temporal Path Proposals
• Convolutional Sequence to Sequence Model for Human Dynamics
• Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networks
• Music Mood Detection Based On Audio And Lyrics With Deep Neural Net
• Cell division in deep material networks applied to multiscale strain localization modeling
• Knowledge as a Teacher: Knowledge-Guided Structural Attention Networks
• Meta-Learning for Low-Resource Neural Machine Translation
• Sentence Pair Scoring: Towards Unified Framework for Text Comprehension
• Transcribing Content from Structural Images with Spotlight Mechanism
• Understanding Hidden Memories of Recurrent Neural Networks
• Online Learning for Neural Machine Translation Post-editing
• Unsupervised temporal context learning using convolutional neural networks for laparoscopic workflow analysis
• Learning to Explain Non-Standard English Words and Phrases
• Efficient Context and Schema Fusion Networks for Multi-Domain Dialogue State Tracking
• Hierarchical Attention-Based Recurrent Highway Networks for Time Series Prediction
• Emotion-Aware Transformer Encoder for Empathetic Dialogue Generation
• General Value Function Networks
• Beyond the C: Retargetable Decompilation using Neural Machine Translation
• Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer
• Efficient Generator of Mathematical Expressions for Symbolic Regression
• Grounded Recurrent Neural Networks
• A comparative study of neural network techniques for automatic software vulnerability detection
• Learning Robust Dialog Policies in Noisy Environments
• Lessons from Natural Language Inference in the Clinical Domain
• Representation Learning for Electronic Health Records
• Stories for Images-in-Sequence by using Visual and Narrative Components
• Memory-augmented Neural Machine Translation
• A Novel Ensemble Deep Learning Model for Stock Prediction Based on Stock Prices and News
• Denoising Distant Supervision for Relation Extraction via Instance-Level Adversarial Training
• A Survey on Recent Advances in Sequence Labeling from Deep Learning Models
• Representation of linguistic form and function in recurrent neural networks
• Neural Networks Models for Entity Discovery and Linking
• Production Ready Chatbots: Generate if not Retrieve
• Neural Personalized Response Generation as Domain Adaptation
• Explaining and Generalizing Back-Translation through Wake-Sleep
• Multi-variable LSTM neural network for autoregressive exogenous model
• Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks
• Toward Goal-Driven Neural Network Models for the Rodent Whisker-Trigeminal System
• What Happened? Leveraging VerbNet to Predict the Effects of Actions in Procedural Text
• Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation
• Automatic Generation of Text Descriptive Comments for Code Blocks
• Verification of Recurrent Neural Networks Through Rule Extraction
• Augmenting Non-Collaborative Dialog Systems with Explicit Semantic and Strategic Dialog History
• Rare Disease Detection by Sequence Modeling with Generative Adversarial Networks
• Predictive analysis of Bitcoin price considering social sentiments
• Bidirectional Recurrent Neural Networks for Medical Event Detection in Electronic Health Records
• Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension
• Generating Natural Language Adversarial Examples on a Large Scale with Generative Models
• Robust Encoder-Decoder Learning Framework towards Offline Handwritten Mathematical Expression Recognition Based on Multi-Scale Deep Neural Network
• Generic Black-Box End-to-End Attack Against State of the Art API Call Based Malware Classifiers
• Improving the Performance of Neural Machine Translation Involving Morphologically Rich Languages
• Improved Regularization Techniques for End-to-End Speech Recognition
• Code Attention: Translating Code to Comments by Exploiting Domain Features
• Neural Machine Translation For Paraphrase Generation
• EdgeRec: Recommender System on Edge in Mobile Taobao
• Structure discovery in Atomic Force Microscopy imaging of ice
• UTRNet: High-Resolution Urdu Text Recognition In Printed Documents
• Context- and Sequence-Aware Convolutional Recurrent Encoder for Neural Machine Translation
• A Comprehensive Survey of Deep Learning for Image Captioning
• Sequence Labeling: A Practical Approach
• Reciprocal Attention Fusion for Visual Question Answering
• Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning
• No Need to Pay Attention: Simple Recurrent Neural Networks Work! (for Answering "Simple" Questions)
• Predicting Movie Genres Based on Plot Summaries
• Automated patent extraction powers generative modeling in focused chemical spaces
• Identification of Gaussian Process State Space Models
• SMT vs NMT: A Comparison over Hindi & Bengali Simple Sentences
• Identifying Harm Events in Clinical Care through Medical Narratives
• A Flexible Approach to Automated RNN Architecture Generation
• Graph Clustering with Dynamic Embedding
• A variational autoencoder for music generation controlled by tonal tension
• Towards Better Opioid Antagonists Using Deep Reinforcement Learning
• A Comparative Study of Rule Extraction for Recurrent Neural Networks
• Automated Bridge Component Recognition using Video Data
• Neural Architectures for Nested NER through Linearization
• Deep Technology Tracing for High-tech Companies
• DTCA: Decision Tree-based Co-Attention Networks for Explainable Claim Verification
• Multimodal Trajectory Prediction via Topological Invariance for Navigation at Uncontrolled Intersections
• Capturing Long-range Contextual Dependencies with Memory-enhanced Conditional Random Fields
• Context, Attention and Audio Feature Explorations for Audio Visual Scene-Aware Dialog
• Literature review on vulnerability detection using NLP technology
• DeepLogic: Towards End-to-End Differentiable Logical Reasoning
• AutoCaption: Image Captioning with Neural Architecture Search
• A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning
• From Words to Sentences: A Progressive Learning Approach for Zero-resource Machine Translation with Visual Pivots
• Temporal Context Aggregation for Video Retrieval with Contrastive Learning
• Multi-task Learning based Pre-trained Language Model for Code Completion
• Medical Dead-ends and Learning to Identify High-risk States and Treatments
• Toward a full-scale neural machine translation in production: the Booking.com use case
• Automated essay scoring using efficient transformer-based language models
• Face Parsing via Recurrent Propagation
• Machine Translation at Booking.com: Journey and Lessons Learned
• Evaluating neural network explanation methods using hybrid documents and morphological agreement
• Recurrent Latent Variable Networks for Session-Based Recommendation
• Towards Neural Decompilation
• CTCModel: a Keras Model for Connectionist Temporal Classification
• Better Sign Language Translation with STMC-Transformer
• Improved Chord Recognition by Combining Duration and Harmonic Language Models
• Log-Linear RNNs: Towards Recurrent Neural Networks with Flexible Prior Knowledge
• Improving End-to-End Speech Recognition with Policy Learning
• Extending Neural Generative Conversational Model using External Knowledge Sources
• Divide-and-Conquer Adversarial Learning for High-Resolution Image and Video Enhancement
• seq2graph: Discovering Dynamic Dependencies from Multivariate Time Series with Multi-level Attention
• Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art
• When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting
• A Survey on Deep Learning for Neuroimaging-based Brain Disorder Analysis
• Time-series Insights into the Process of Passing or Failing Online University Courses using Neural-Induced Interpretable Student States
• Getting deep recommenders fit: Bloom embeddings for sparse binary input/output networks
• DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks
• Revisiting the Design Issues of Local Models for Japanese Predicate-Argument Structure Analysis
• Automated proof synthesis for propositional logic with deep neural networks
• BanglaBait: Semi-Supervised Adversarial Approach for Clickbait Detection on Bangla Clickbait Dataset
• Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioning
• Probabilistic Embeddings for Cross-Modal Retrieval
• External Correlates of Adult Digital Problem-Solving Behavior: Log Data Analysis of a Large-Scale Assessment
• An Efficient Character-Level Neural Machine Translation
• Imitation Learning for Human Pose Prediction
• Explainable Multivariate Time Series Classification: A Deep Neural Network Which Learns To Attend To Important Variables As Well As Informative Time Intervals
• MixSeq: Connecting Macroscopic Time Series Forecasting with Microscopic Time Series Data
• Towards Solving Text-based Games by Producing Adaptive Action Spaces
• ARAML: A Stable Adversarial Training Framework for Text Generation
• Scene Labeling using Gated Recurrent Units with Explicit Long Range Conditioning
• Optimal Mini-Batch Size Selection for Fast Gradient Descent
• Predictive Coding for Boosting Deep Reinforcement Learning with Sparse Rewards
• Tensorized Transformer for Dynamical Systems Modeling
• Fusing Recency into Neural Machine Translation with an Inter-Sentence Gate Model
• Recurrent One-Hop Predictions for Reasoning over Knowledge Graphs
• Back2Future: Leveraging Backfill Dynamics for Improving Real-time Predictions in Future
• Autoencoder Regularized Network For Driving Style Representation Learning
• Variational online learning of neural dynamics
• Improving historical spelling normalization with bi-directional LSTMs and multi-task learning
• Visual Fashion-Product Search at SK Planet
• Read, Tag, and Parse All at Once, or Fully-neural Dependency Parsing
• Multilingual Neural Machine Translation for Zero-Resource Languages
• Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP
• Memory Visualization for Gated Recurrent Neural Networks in Speech Recognition
• Polyphone Disambiguation for Mandarin Chinese Using Conditional Neural Network with Multi-level Embedding Features
• Enhancing Salient Object Segmentation Through Attention
• One Size Does Not Fit All: Multi-Scale, Cascaded RNNs for Radar Classification
• Why and How to Pay Different Attention to Phrase Alignments of Different Intensities
• Empirical Evaluation of RNN Architectures on Sentence Classification Task
• Neural Machine Translation with Source-Side Latent Graph Parsing
• CryptoAnalytics: Cryptocoins Price Forecasting with Machine Learning Techniques
• Interpreting the Syntactic and Social Elements of the Tweet Representations via Elementary Property Prediction Tasks
• Spatial-Temporal Tensor Graph Convolutional Network for Traffic Prediction
• CNNs, LSTMs, and Attention Networks for Pathology Detection in Medical Data
• Deep Animation Video Interpolation in the Wild
• Rapid Spectral Parameter Prediction for Black Hole X-Ray Binaries using Physicalised Autoencoders
• Analogs of Linguistic Structure in Deep Representations
• Towards Socially Intelligent Agents with Mental State Transition and Human Utility
• Extending a model for ontology-based Arabic-English machine translation
• EDSL: An Encoder-Decoder Architecture with Symbol-Level Features for Printed Mathematical Expression Recognition
• Self-Supervised Learning and Prediction of Microstructure Evolution with Recurrent Neural Networks
• Patching as Translation: the Data and the Metaphor
• Y^2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences
• Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory
• A Hierarchical Deep Convolutional Neural Network and Gated Recurrent Unit Framework for Structural Damage Detection
• Session-Based Recommender Systems for Action Selection in GUI Test Generation
• Looking for change? Roll the Dice and demand Attention
• Multilevel Text Normalization with Sequence-to-Sequence Networks and Multisource Learning
• CompNet: Neural networks growing via the compact network morphism
• Structured Deep Kernel Networks for Data-Driven Closure Terms of Turbulent Flows
• Machine Learning Applications for Therapeutic Tasks with Genomics Data
• Enhanced Neural Machine Translation by Learning from Draft
• Rotational Unit of Memory
• Towards Music Captioning: Generating Music Playlist Descriptions
• Sampling-based Gradient Regularization for Capturing Long-Term Dependencies in Recurrent Neural Networks
• A Variational Auto-Encoder for Reservoir Monitoring
• Explanations for Temporal Recommendations
• Dynamic Graph Generation Network: Generating Relational Knowledge from Diagrams
• Victim or Perpetrator? Analysis of Violent Characters Portrayals from Movie Scripts
• MagnifierNet: Towards Semantic Adversary and Fusion for Person Re-identification
• GeoSeq2Seq: Information Geometric Sequence-to-Sequence Networks
• A Deep Structural Model for Analyzing Correlated Multivariate Time Series
• Language Transfer of Audio Word2Vec: Learning Audio Segment Representations without Target Language Data
• Online Second Order Methods for Non-Convex Stochastic Optimizations
• Paraphrase Detection on Noisy Subtitles in Six Languages
• Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation
• Learning Dynamical Systems from Noisy Sensor Measurements using Multiple Shooting
• An Optimized and Energy-Efficient Parallel Implementation of Non-Iteratively Trained Recurrent Neural Networks
• Memory and attention in deep learning
• Language Modeling with Generative Adversarial Networks
• Recursive Top-Down Production for Sentence Generation with Latent Trees
• Interactive Fiction Game Playing as Multi-Paragraph Reading Comprehension with Reinforcement Learning
• Latent Variable Algorithms for Multimodal Learning and Sensor Fusion
• Let's See Clearly: Contaminant Artifact Removal for Moving Cameras
• Maintaining Common Ground in Dynamic Environments
• Scalable Machine Translation in Memory Constrained Environments
• An End-to-End Approach to Automatic Speech Assessment for Cantonese-speaking People with Aphasia
• Highway State Gating for Recurrent Highway Networks: improving information flow through time
• A Forecasting System of Computational Time of DFT/TDDFT Calculations under the Multiverse ansatz via Machine Learning and Cheminformatics
• Plan, Attend, Generate: Character-level Neural Machine Translation with Planning in the Decoder
• Structured Prediction in NLP -- A survey
• Using GGNN to recommend log statement level
• Efficient Global String Kernel with Random Features: Beyond Counting Substructures
• Efficient Purely Convolutional Text Encoding
• Multi-Step Chord Sequence Prediction Based on Aggregated Multi-Scale Encoder-Decoder Network
• Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
• Shapley Homology: Topological Analysis of Sample Influence for Neural Networks
• Applying recent advances in Visual Question Answering to Record Linkage
• Enhancing Model Robustness By Incorporating Adversarial Knowledge Into Semantic Representation
• Data-Efficient Methods for Dialogue Systems
• LSTM-CF: Unifying Context Modeling and Fusion with LSTMs for RGB-D Scene Labeling
• The RLLChatbot: a solution to the ConvAI challenge
• Attentive Long Short-Term Preference Modeling for Personalized Product Search
• Czech Text Processing with Contextual Embeddings: POS Tagging, Lemmatization, Parsing and NER
• Combining Word and Character Vector Representation on Neural Machine Translation
• Learning in Gated Neural Networks