An Empirical Study of Smoothing Techniques for Language Modeling

Abstract

We present an extensive empirical comparison of several smoothing techniques in the domain of language modeling, including those described by Jelinek and Mercer (1980), Katz (1987), and Church and Gale (1991). We investigate for the first time how factors such as training data size, corpus (e.g., Brown versus Wall Street Journal), and n-gram order (bigram versus trigram) affect the relative performance of these methods, which we measure through the cross-entropy of test data. In addition, we introduce two novel smoothing techniques, one a variation of Jelinek-Mercer smoothing and one a very simple linear interpolation technique, both of which outperform existing methods.

9 pages, LaTeX, uses aclap.sty

References in corpus (2)

• Natural Language Parsing as Statistical Pattern Recognition
• Building Probabilistic Models for Natural Language

Cited by in corpus (62)

• Language Modeling with Gated Convolutional Networks
• Finding Deceptive Opinion Spam by Any Stretch of the Imagination
• Learning Natural Coding Conventions
• A Survey of Location Prediction on Twitter
• Learning to Resolve Natural Language Ambiguities: A Unified Approach
• Data Noising as Smoothing in Neural Network Language Models
• Aggregate and mixed-order Markov models for statistical language processing
• Pretrained Transformers for Text Ranking: BERT and Beyond
• Building Probabilistic Models for Natural Language
• Similarity-Based Approaches to Natural Language Processing
• Similarity-Based Models of Word Cooccurrence Probabilities
• Neural Networks for Text Correction and Completion in Keyboard Decoding
• An Empirical Comparison of Probability Models for Dependency Grammar
• Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT2015
• Natural Language Understanding with Distributed Representation
• Maybe Deep Neural Networks are the Best Choice for Modeling Source Code
• Larger-Context Language Modelling
• A Winnow-Based Approach to Context-Sensitive Spelling Correction
• Montage: A Neural Network Language Model-Guided JavaScript Engine Fuzzer
• Comparing Neural- and N-Gram-Based Language Models for Word Segmentation
• Efficient Transfer Learning Schemes for Personalized Language Modeling using Recurrent Neural Network
• Reordering rules for English-Hindi SMT
• Using Syntax-Based Machine Translation to Parse English into Abstract Meaning Representation
• Semi-Supervised Model Training for Unbounded Conversational Speech Recognition
• Visual Text Correction
• Speech Repairs, Intonational Boundaries and Discourse Markers: Modeling Speakers' Utterances in Spoken Dialog
• Using Google Ngram Viewer for Scientific Referencing and History of Science
• $gen$CNN: A Convolutional Architecture for Word Sequence Prediction
• Distributional Inclusion Hypothesis for Tensor-based Composition
• Utility-Theoretic Ranking for Semi-Automated Text Classification
• Language Detection For Short Text Messages In Social Media
• Prediction with a Short Memory
• A Survey On Neural Word Embeddings
• Building a reordering system using tree-to-string hierarchical model
• Teaching Machines to Converse
• Progress and Tradeoffs in Neural Language Models
• A Re-ranker Scheme for Integrating Large Scale NLU models
• TRUST-TECH based Methods for Optimization and Learning
• Accurate Local Estimation of Geo-Coordinates for Social Media Posts
• Variational Smoothing in Recurrent Neural Network Language Models
• Direct Output Connection for a High-Rank Language Model
• Learning Autocompletion from Real-World Datasets
• Comparison of Modified Kneser-Ney and Witten-Bell Smoothing Techniques in Statistical Language Model of Bahasa Indonesia
• Machine Translation in Indian Languages: Challenges and Resolution
• Character n-gram Embeddings to Improve RNN Language Models
• Joint Space Neural Probabilistic Language Model for Statistical Machine Translation
• Improving Quality of Hierarchical Clustering for Large Data Series
• Understanding Common Perceptions from Online Social Media
• Enhancing Automatically Discovered Multi-level Acoustic Patterns Considering Context Consistency With Applications in Spoken Term Detection
• Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation
• Neural and Statistical Methods for Leveraging Meta-information in Machine Translation
• The Structurally Smoothed Graphlet Kernel
• On Training Bi-directional Neural Network Language Model with Noise Contrastive Estimation
• Estimating Confusions in the ASR Channel for Improved Topic-based Language Model Adaptation
• Dependency Graph-to-String Statistical Machine Translation
• On the Development of Text Input Method - Lessons Learned
• Dialog Context Language Modeling with Recurrent Neural Networks
• Neural or Statistical: An Empirical Study on Language Models for Chinese Input Recommendation on Mobile
• TransfoRNN: Capturing the Sequential Information in Self-Attention Representations for Language Modeling
• Unsupervised Meta Learning for One Shot Title Compression in Voice Commerce
• SAM: Semantic Attribute Modulation for Language Modeling and Style Variation
• On Modeling Profiles instead of Values