Improving Neural Language Modeling via Adversarial Training

Abstract

Recently, substantial progress has been made in language modeling by using deep neural networks. However, in practice, large scale neural language models have been shown to be prone to overfitting. In this paper, we present a simple yet highly effective adversarial training mechanism for regularizing neural language models. The idea is to introduce adversarial noise to the output embedding layer while training the models. We show that the optimal adversarial noise yields a simple closed-form solution, thus allowing us to develop a simple and time efficient algorithm. Theoretically, we show that our adversarial mechanism effectively encourages the diversity of the embedding vectors, helping to increase the robustness of models. Empirically, we show that our method improves on the single model state-of-the-art results for language modeling on Penn Treebank (PTB) and Wikitext-2, achieving test perplexity scores of 46.01 and 38.07, respectively. When applied to machine translation, our method improves over various transformer-based translation baselines in BLEU scores on the WMT14 English-German and IWSLT14 German-English tasks.

Cited by in corpus (24)

• AutoML: A Survey of the State-of-the-Art
• A Survey on Data Augmentation for Text Classification
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Adversarial Training for Large Neural Language Models
• Neural Language Generation: Formulation, Methods, and Evaluation
• Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
• Adversarially trained LSTMs on reduced order models of urban air pollution simulations
• Towards Security Threats of Deep Learning Systems: A Survey
• Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning
• Adversarial Feature Augmentation and Normalization for Visual Recognition
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Rethinking Perturbations in Encoder-Decoders for Fast Training
• Adversarial Attack and Defense of Structured Prediction Models
• CAP: Co-Adversarial Perturbation on Weights and Features for Improving Generalization of Graph Neural Networks
• Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches
• IOT: Instance-wise Layer Reordering for Transformer Structures
• BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translation
• Posterior Differential Regularization with f-divergence for Improving Model Robustness
• Noisy Training Improves E2E ASR for the Edge
• UniDrop: A Simple yet Effective Technique to Improve Transformer without Extra Cost
• An empirical analysis of phrase-based and neural machine translation
• ARCH: Efficient Adversarial Regularized Training with Caching
• Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
• Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models