Data Noising as Smoothing in Neural Network Language Models

Abstract

Data noising is an effective technique for regularizing neural network models. While noising is widely adopted in application domains such as vision and speech, commonly used noising primitives have not been developed for discrete sequence-level settings such as language modeling. In this paper, we derive a connection between input noising in neural network language models and smoothing in $n$-gram models. Using this connection, we draw upon ideas from smoothing to develop effective noising schemes. We demonstrate performance gains when applying the proposed schemes to language modeling and machine translation. Finally, we provide empirical analysis validating the relationship between noising and smoothing.

ICLR 2017

References in corpus (11)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Recurrent Neural Network Regularization
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• An Empirical Study of Smoothing Techniques for Language Modeling
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Pointer Sentinel Mixture Models
• Unitary Evolution Recurrent Neural Networks
• Recurrent Dropout without Memory Loss
• Regularizing RNNs by Stabilizing Activations

Cited by in corpus (41)

• AutoML: A Survey of the State-of-the-Art
• A Survey on Data Augmentation for Text Classification
• Data Augmentation Approaches in Natural Language Processing: A Survey
• EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
• Neural Text Generation: A Practical Guide
• Improving Variational Encoder-Decoders in Dialogue Generation
• DeepSentiPers: Novel Deep Learning Models Trained Over Proposed Augmented Persian Sentiment Corpus
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• GNN-LM: Language Modeling based on Global Contexts via GNN
• Denoising Distant Supervision for Relation Extraction via Instance-Level Adversarial Training
• Integrating Semantic Knowledge to Tackle Zero-shot Text Classification
• DropDim: A Regularization Method for Transformer Networks
• Conditional BERT Contextual Augmentation
• Robust Multilingual Part-of-Speech Tagging via Adversarial Training
• Pushing the bounds of dropout
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks
• Influence-guided Data Augmentation for Neural Tensor Completion
• Connecting the Dots Between MLE and RL for Sequence Prediction
• Semantic Equivalent Adversarial Data Augmentation for Visual Question Answering
• How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• Cost-Sensitive BERT for Generalisable Sentence Classification with Imbalanced Data
• Data Augmentation for Neural Online Chat Response Selection
• Improving Robustness and Generality of NLP Models Using Disentangled Representations
• OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts
• Progressive Generation of Long Text with Pretrained Language Models
• Approaching Neural Chinese Word Segmentation as a Low-Resource Machine Translation Task
• Reducing and Exploiting Data Augmentation Noise through Meta Reweighting Contrastive Learning for Text Classification
• Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweight
• Variational Smoothing in Recurrent Neural Network Language Models
• A little goes a long way: Improving toxic language classification despite data scarcity
• Goal-directed Generation of Discrete Structures with Conditional Generative Models
• Learning to Augment for Data-Scarce Domain BERT Knowledge Distillation
• Adversarial Learning for Supervised and Semi-supervised Relation Extraction in Biomedical Literature
• Uncertainty-Aware Semantic Augmentation for Neural Machine Translation
• Improved Dynamic Memory Network for Dialogue Act Classification with Adversarial Training
• Token Drop mechanism for Neural Machine Translation
• N-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking
• PHICON: Improving Generalization of Clinical Text De-identification Models via Data Augmentation
• Imbalanced Sentiment Classification Enhanced with Discourse Marker
• Towards Efficiently Diversifying Dialogue Generation via Embedding Augmentation