Markov Chain Monte Carlo for generating ranked textual data

Abstract

This paper faces a central theme in applied statistics and information science, which is the assessment of the stochastic structure of rank-size laws in text analysis. We consider the words in a corpus by ranking them on the basis of their frequencies in descending order. The starting point is that the ranked data generated in linguistic contexts can be viewed as the realisations of a discrete states Markov chain, whose stationary distribution behaves according to a discretisation of the best fitted rank-size law. The employed methodological toolkit is Markov Chain Monte Carlo, specifically referring to the Metropolis-Hastings algorithm. The theoretical framework is applied to the rank-size analysis of the hapax legomena occurring in the speeches of the US Presidents. We offer a large number of statistical tests leading to the consistency of our methodological proposal. To pursue our scopes, we also offer arguments supporting that hapaxes are rare (``extreme") events resulting from memory-less-like processes. Moreover, we show that the considered sample has the stochastic structure of a Markov chain of order one. Importantly, we discuss the versatility of the method, which is considered suitable for deducing similar outcomes for other applied science contexts.

References in corpus (17)

• Power-law distributions in binned empirical data
• On Prediction Using Variable Order Markov Models
• A Survey of Monte Carlo Methods for Parameter Estimation
• A Review of Multiple Try MCMC algorithms for Signal Processing
• A statistical framework for testing functional categories in microarray data
• Evidence of Economic Regularities and Disparities of Italian Regions From Aggregated Tax Income Size Data
• A universal rank-size law
• Longitudinal high-dimensional principal components analysis with application to diffusion tensor imaging of multiple sclerosis
• A joint text mining-rank size investigation of the rhetoric structures of the US Presidents' speeches
• Some things we've learned (about Markov chain Monte Carlo)
• Explaining Zipf's Law via Mental Lexicon
• Punctuation effects in English and Esperanto texts
• Simon's fundamental rich-get-richer model entails a dominant first-mover advantage
• Earthquakes economic costs through rank-size laws
• Words ranking and Hirsch index for identifying the core of the hapaxes in political texts
• Effect of memory in non-Markovian Boolean networks
• Coherent measures of the impact of co-authors in peer review journals and in proceedings publications