Neural Machine Translation in Linear Time

Abstract

We present a novel neural network for processing sequences. The ByteNet is a one-dimensional convolutional neural network that is composed of two parts, one to encode the source sequence and the other to decode the target sequence. The two network parts are connected by stacking the decoder on top of the encoder and preserving the temporal resolution of the sequences. To address the differing lengths of the source and the target, we introduce an efficient mechanism by which the decoder is dynamically unfolded over the representation of the encoder. The ByteNet uses dilation in the convolutional layers to increase its receptive field. The resulting network has two core properties: it runs in time that is linear in the length of the sequences and it sidesteps the need for excessive memorization. The ByteNet decoder attains state-of-the-art performance on character-level language modelling and outperforms the previous best results obtained with recurrent networks. The ByteNet also achieves state-of-the-art performance on character-to-character machine translation on the English-to-German WMT translation task, surpassing comparable neural translation models that are based on recurrent networks with attentional pooling and run in quadratic time. We find that the latent alignment structure contained in the representations reflects the expected alignment between the tokens.

9 pages

References in corpus (2)

• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Hierarchical Multiscale Recurrent Neural Networks

Cited by in corpus (133)

• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• Non-Autoregressive Neural Machine Translation
• Aggregated Residual Transformations for Deep Neural Networks
• Quasi-Recurrent Neural Networks
• Recent Advances in Convolutional Neural Networks
• Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
• How to Build a Graph-Based Deep Learning Architecture in Traffic Domain: A Survey
• Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with NBEATSx
• One Model To Learn Them All
• Depthwise Separable Convolutions for Neural Machine Translation
• Coronary Artery Centerline Extraction in Cardiac CT Angiography Using a CNN-Based Orientation Classifier
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Weighted Transformer Network for Machine Translation
• Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
• Generating and designing DNA with deep generative models
• Understanding Back-Translation at Scale
• eXpose: A Character-Level Convolutional Neural Network with Embeddings For Detecting Malicious URLs, File Paths and Registry Keys
• A Survey of Deep Learning Techniques for Neural Machine Translation
• Generative Models for Automatic Chemical Design
• Trellis Networks for Sequence Modeling
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• Multiplicative LSTM for sequence modelling
• Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling
• Graph WaveNet for Deep Spatial-Temporal Graph Modeling
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• Learning Sparse Networks Using Targeted Dropout
• Efficient Neural Audio Synthesis
• Gated-Dilated Networks for Lung Nodule Classification in CT scans
• CNN+CNN: Convolutional Decoders for Image Captioning
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Dynamic Evaluation of Neural Sequence Models
• Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction
• BP-Transformer: Modelling Long-Range Context via Binary Partitioning
• Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
• Towards Accurate Scene Text Recognition with Semantic Reasoning Networks
• Beyond exploding and vanishing gradients: analysing RNN training using attractors and smoothness
• Compressive Transformers for Long-Range Sequence Modelling
• Parallel Multiscale Autoregressive Density Estimation
• Dual Rectified Linear Units (DReLUs): A Replacement for Tanh Activation Functions in Quasi-Recurrent Neural Networks
• A Hybrid Convolutional Variational Autoencoder for Text Generation
• Deep Learning Based Chatbot Models
• Neural Text Generation: A Practical Guide
• Multi-range Reasoning for Machine Comprehension
• Data-driven Summarization of Scientific Articles
• Convolutional Sequence to Sequence Model for Human Dynamics
• Boosting Neural Machine Translation
• Unsupervised Cipher Cracking Using Discrete GANs
• Parallelizing Linear Recurrent Neural Nets Over Sequence Length
• On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
• FastWave: Accelerating Autoregressive Convolutional Neural Networks on FPGA
• Explainable artificial intelligence model to predict acute critical illness from electronic health records
• Neural Language Generation: Formulation, Methods, and Evaluation
• You May Not Need Attention
• Extreme Multi-Label Legal Text Classification: A case study in EU Legislation
• Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositions
• Fast Generation for Convolutional Autoregressive Models
• Towards Robust Neural Machine Translation
• Neural message passing for joint paratope-epitope prediction
• Time-aware Large Kernel Convolutions
• TTS-Portuguese Corpus: a corpus for speech synthesis in Brazilian Portuguese
• Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input
• Do Neural Nets Learn Statistical Laws behind Natural Language?
• Trainable Greedy Decoding for Neural Machine Translation
• Tensor2Tensor for Neural Machine Translation
• Temporal Deformable Convolutional Encoder-Decoder Networks for Video Captioning
• Hybrid Self-Attention Network for Machine Translation
• Hard-Coded Gaussian Attention for Neural Machine Translation
• Finite Volume Neural Network: Modeling Subsurface Contaminant Transport
• How to Hallucinate Functional Proteins
• Deep Neural Machine Translation with Weakly-Recurrent Units
• Hint-Based Training for Non-Autoregressive Machine Translation
• Sequential Deep Learning for Credit Risk Monitoring with Tabular Financial Data
• LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary Attention
• 1D Convolutional Neural Network Models for Sleep Arousal Detection
• Phrase-Based Attentions
• Joint Echo Cancellation and Noise Suppression based on Cascaded Magnitude and Complex Mask Estimation
• Neural Machine Translation: A Review of Methods, Resources, and Tools
• CNN Is All You Need
• Better Sign Language Translation with STMC-Transformer
• Future Data Helps Training: Modeling Future Contexts for Session-based Recommendation
• Echo: Compiler-based GPU Memory Footprint Reduction for LSTM RNN Training
• Morphological Inflection Generation with Hard Monotonic Attention
• Parallel Grid Pooling for Data Augmentation
• Anytime Sampling for Autoregressive Models via Ordered Autoencoding
• Hierarchical Sequence to Sequence Voice Conversion with Limited Data
• Discovery of Natural Language Concepts in Individual Units of CNNs
• Spatial PixelCNN: Generating Images from Patches
• On the Blindspots of Convolutional Networks
• User-specific Adaptive Fine-tuning for Cross-domain Recommendations
• Universal Approximation of Input-Output Maps by Temporal Convolutional Nets
• Teaching Machines to Code: Neural Markup Generation with Visual Attention
• Skillful Twelve Hour Precipitation Forecasts using Large Context Neural Networks
• A Simple Convolutional Generative Network for Next Item Recommendation
• The TALP-UPC System for the WMT Similar Language Task: Statistical vs Neural Machine Translation
• DTMT: A Novel Deep Transition Architecture for Neural Machine Translation
• Semantic-Unit-Based Dilated Convolution for Multi-Label Text Classification
• Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism
• Multi-Task Gaussian Processes and Dilated Convolutional Networks for Reconstruction of Reproductive Hormonal Dynamics
• Efficient Bidirectional Neural Machine Translation
• Dynamic Sentence Sampling for Efficient Training of Neural Machine Translation
• Encoding Spatial Relations from Natural Language
• A Feasible Framework for Arbitrary-Shaped Scene Text Recognition
• Enhanced 3D Human Pose Estimation from Videos by using Attention-Based Neural Network with Dilated Convolutions
• Recurrent Point Review Models
• EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators
• Neural Machine Translation: A Review and Survey
• Extending Recurrent Neural Aligner for Streaming End-to-End Speech Recognition in Mandarin
• Neural Contract Element Extraction Revisited: Letters from Sesame Street
• Upgrading the Newsroom: An Automated Image Selection System for News Articles
• Formant Tracking Using Dilated Convolutional Networks Through Dense Connection with Gating Mechanism
• CytonMT: an Efficient Neural Machine Translation Open-source Toolkit Implemented in C++
• DsMCL: Dual-Level Stochastic Multiple Choice Learning for Multi-Modal Trajectory Prediction
• Long-Term Human Video Generation of Multiple Futures Using Poses
• Cut-Based Graph Learning Networks to Discover Compositional Structure of Sequential Video Data
• Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN
• Nonsymbolic Text Representation
• Machine Translation between Vietnamese and English: an Empirical Study
• Mining Commonsense Facts from the Physical World
• Sentence-wise Smooth Regularization for Sequence to Sequence Learning
• Question-Aware Sentence Gating Networks for Question and Answering
• Understanding Feature Selection and Feature Memorization in Recurrent Neural Networks
• Attentive Sequence to Sequence Translation for Localizing Clips of Interest by Natural Language Descriptions
• Future-Prediction-Based Model for Neural Machine Translation
• Self-Attentional Models Application in Task-Oriented Dialogue Generation Systems
• Cross Copy Network for Dialogue Generation
• Developing neural machine translation models for Hungarian-English
• AI-lead Court Debate Case Investigation
• 3D human pose estimation with adaptive receptive fields and dilated temporal convolutions
• Water Supply Prediction Based on Initialized Attention Residual Network
• Temporally Folded Convolutional Neural Networks for Sequence Forecasting
• Back to the Future: Joint Aware Temporal Deep Learning 3D Human Pose Estimation