Empirical Analysis of Zipf's Law, Power Law, and Lognormal Distributions in Medical Discharge Reports

Abstract

Bayesian modelling and statistical text analysis rely on informed probability priors to encourage good solutions. This paper empirically analyses whether text in medical discharge reports follow Zipf's law, a commonly assumed statistical property of language where word frequency follows a discrete power law distribution. We examined 20,000 medical discharge reports from the MIMIC-III dataset. Methods included splitting the discharge reports into tokens, counting token frequency, fitting power law distributions to the data, and testing whether alternative distributions--lognormal, exponential, stretched exponential, and truncated power law--provided superior fits to the data. Results show that discharge reports are best fit by the truncated power law and lognormal distributions. Our findings suggest that Bayesian modelling and statistical text analysis of discharge report text would benefit from using truncated power law and lognormal probability priors.

arXiv commentReduced word count to 3000, moved methods details to appendices

References in corpus (10)

• Power-law distributions in empirical data
• Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis
• Powerlaw: a Python package for analysis of heavy-tailed distributions
• A network approach to topic models
• A Review of Challenges and Opportunities in Machine Learning for Health
• Large-scale analysis of Zipf's law in English texts
• Fitting heavy tailed distributions: the poweRlaw package
• Zipf's law for word frequencies: word forms versus lemmas in long texts
• Nonparametric Bayesian Topic Modelling with the Hierarchical Pitman-Yor Processes
• Beyond the Chinese Restaurant and Pitman-Yor processes: Statistical Models with Double Power-law Behavior