Aggregate and mixed-order Markov models for statistical language processing

Abstract

We consider the use of language models whose size and accuracy are intermediate between different order n-gram models. Two types of models are studied in particular. Aggregate Markov models are class-based bigram models in which the mapping from words to classes is probabilistic. Mixed-order Markov models combine bigram models whose predictions are conditioned on different words. Both types of models are trained by Expectation-Maximization (EM) algorithms for maximum likelihood estimation. We examine smoothing procedures in which these models are interposed between different order n-grams. This is found to significantly reduce the perplexity of unseen word combinations.

9 pages, 4 PostScript figures, uses psfig.sty and aclap.sty; to appear in the proceedings of EMNLP-2

References in corpus (1)

• An Empirical Study of Smoothing Techniques for Language Modeling

Cited by in corpus (9)

• Probabilistic Latent Semantic Analysis
• Spectral Unmixing via Data-guided Sparsity
• Structured Generative Models of Natural Source Code
• Similarity-Based Models of Word Cooccurrence Probabilities
• Search Using N-gram Technique Based Statistical Analysis for Knowledge Extraction in Case Based Reasoning Systems
• Structured Sparse Method for Hyperspectral Unmixing
• Modeling Online Reviews with Multi-grain Topic Models
• A Bit of Progress in Language Modeling
• Probabilistic top-down parsing and language modeling