On Using Monolingual Corpora in Neural Machine Translation

Abstract

Recent work on end-to-end neural network-based architectures for machine translation has shown promising results for En-Fr and En-De translation. Arguably, one of the major factors behind this success has been the availability of high quality parallel corpora. In this work, we investigate how to leverage abundant monolingual corpora for neural machine translation. Compared to a phrase-based and hierarchical baseline, we obtain up to $1.96$ BLEU improvement on the low-resource language pair Turkish-English, and $1.59$ BLEU on the focused domain task of Chinese-English chat messages. While our method was initially targeted toward such tasks with less parallel data, we show that it also extends to high resource languages such as Cs-En and De-En where we obtain an improvement of $0.39$ and $0.47$ BLEU scores over the neural machine translation baselines, respectively.

9 pages, 2 figures

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Theano: new features and speed improvements
• Addressing the Rare Word Problem in Neural Machine Translation

Cited by in corpus (128)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Attention-Based Models for Speech Recognition
• Dual Learning for Machine Translation
• Self-Supervised Speech Representation Learning: A Review
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Improved training of end-to-end attention models for speech recognition
• Improved Noisy Student Training for Automatic Speech Recognition
• Phrase-Based & Neural Unsupervised Machine Translation
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition
• Unsupervised Machine Translation Using Monolingual Corpora Only
• A Survey of Available Corpora for Building Data-Driven Dialogue Systems
• Language Modeling with Deep Transformers
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Neural Networks with Few Multiplications
• Neural Summarization by Extracting Sentences and Words
• A Neural Knowledge Language Model
• Mutual Information and Diverse Decoding Improve Neural Machine Translation
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Sample Efficient Text Summarization Using a Single Pre-Trained Transformer
• Unsupervised Translation of Programming Languages
• Neural Machine Translation with Reconstruction
• Soft Contextual Data Augmentation for Neural Machine Translation
• Towards better decoding and language model integration in sequence to sequence models
• Learning to Decode for Future Success
• Unsupervised Text Style Transfer using Language Models as Discriminators
• Bridging Neural Machine Translation and Bilingual Dictionaries
• Minimum Risk Training for Neural Machine Translation
• RescoreBERT: Discriminative Speech Recognition Rescoring with BERT
• Search Engine Guided Non-Parametric Neural Machine Translation
• Simulated annealing for optimization of graphs and sequences
• Neural Machine Translation with Pivot Languages
• Joint Training for Neural Machine Translation Models with Monolingual Data
• Semi-Supervised QA with Generative Domain-Adaptive Nets
• Exploiting Neural Query Translation into Cross Lingual Information Retrieval
• Meta-Learning for Low-Resource Neural Machine Translation
• Pre-trained Language Model Representations for Language Generation
• On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
• Encoder-Agnostic Adaptation for Conditional Language Generation
• SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression
• Unsupervised pre-training for sequence to sequence speech recognition
• Unsupervised Pretraining for Sequence to Sequence Learning
• Neural Machine Translation with Monolingual Translation Memory
• Can Unconditional Language Models Recover Arbitrary Sentences?
• Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
• Efficient Training of Neural Transducer for Speech Recognition
• Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model
• A Survey of Methods to Leverage Monolingual Data in Low-resource Neural Machine Translation
• On-Device Personalization of Automatic Speech Recognition Models for Disordered Speech
• Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
• An investigation of phone-based subword units for end-to-end speech recognition
• Acquiring Knowledge from Pre-trained Model to Neural Machine Translation
• Improving Sign Language Translation with Monolingual Data by Sign Back-Translation
• Neural Machine Translation: Challenges, Progress and Future
• End-to-end Speech Recognition with Word-based RNN Language Models
• end-to-end training of a large vocabulary end-to-end speech recognition system
• Incorporating Bilingual Dictionaries for Low Resource Semi-Supervised Neural Machine Translation
• Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT
• Improving Domain Adaptation Translation with Domain Invariant and Specific Information
• On Synthetic Data for Back Translation
• Improving accuracy of rare words for RNN-Transducer through unigram shallow fusion
• Machine Translation : From Statistical to modern Deep-learning practices
• Application of Low-resource Machine Translation Techniques to Russian-Tatar Language Pair
• End2End Acoustic to Semantic Transduction
• Domain Adaptation of Neural Machine Translation by Lexicon Induction
• Large-Scale Self- and Semi-Supervised Learning for Speech Translation
• Multimodal Conditionality for Natural Language Generation
• Robust Beam Search for Encoder-Decoder Attention Based Speech Recognition without Length Bias
• The Effect of Domain and Diacritics in Yorùbá-English Neural Machine Translation
• Unsupervised Paraphrasing by Simulated Annealing
• Transfer learning of language-independent end-to-end ASR with language model fusion
• Contextual Density Ratio for Language Model Biasing of Sequence to Sequence ASR Systems
• Improving Neural Machine Translation with Pre-trained Representation
• Attentional Speech Recognition Models Misbehave on Out-of-domain Utterances
• Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability
• Candidate Fusion: Integrating Language Modelling into a Sequence-to-Sequence Handwritten Word Recognition Architecture
• Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition
• Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition
• AR: Auto-Repair the Synthetic Data for Neural Machine Translation
• Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation
• SpecAugment on Large Scale Datasets
• Injecting Text in Self-Supervised Speech Pretraining
• Distilling the Knowledge of BERT for Sequence-to-Sequence ASR
• Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion
• End-to-end attention-based distant speech recognition with Highway LSTM
• Decomposable Neural Paraphrase Generation
• Residual Energy-Based Models for End-to-End Speech Recognition
• Discovering Useful Sentence Representations from Large Pretrained Language Models
• Overcoming Domain Mismatch in Low Resource Sequence-to-Sequence ASR Models using Hybrid Generated Pseudotranscripts
• BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?
• Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition
• Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages
• Meta Back-translation
• Language model integration based on memory control for sequence to sequence speech recognition
• Iterative Domain-Repaired Back-Translation
• How Do Source-side Monolingual Word Embeddings Impact Neural Machine Translation?
• A review of on-device fully neural end-to-end automatic speech recognition algorithms
• A Study of Multilingual End-to-End Speech Recognition for Kazakh, Russian, and English
• Language Models not just for Pre-training: Fast Online Neural Noisy Channel Modeling
• A spelling correction model for end-to-end speech recognition
• Neural Machine Translation: A Review and Survey
• Towards the Unseen: Iterative Text Recognition by Distilling from Errors
• Improved Multi-Stage Training of Online Attention-based Encoder-Decoder Models
• NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21
• Infusing Future Information into Monotonic Attention Through Language Models
• Code Generation from Natural Language with Less Prior and More Monolingual Data
• ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization
• Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer
• Context-aware Decoder for Neural Machine Translation using a Target-side Document-Level Language Model
• Phoneme Based Neural Transducer for Large Vocabulary Speech Recognition
• On Minimum Word Error Rate Training of the Hybrid Autoregressive Transducer
• Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systems
• Deep Shallow Fusion for RNN-T Personalization
• Domain Adaptation of NMT models for English-Hindi Machine Translation Task at AdapMT ICON 2020
• Audio-Visual Decision Fusion for WFST-based and seq2seq Models
• Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition
• Large Margin Neural Language Model
• Language Model-Driven Unsupervised Neural Machine Translation
• Leveraging Text Data Using Hybrid Transformer-LSTM Based End-to-End ASR in Transfer Learning
• Zero-Shot Language Transfer vs Iterative Back Translation for Unsupervised Machine Translation
• Dynamic Fusion: Attentional Language Model for Neural Machine Translation
• Integrating Knowledge into End-to-End Speech Recognition from External Text-Only Data
• Reciprocal Supervised Learning Improves Neural Machine Translation
• Context-Aware Transformer Transducer for Speech Recognition
• Causal Direction of Data Collection Matters: Implications of Causal and Anticausal Learning for NLP
• Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition
• Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT
• Cross-Modal Transformer-Based Neural Correction Models for Automatic Speech Recognition