Statistical keyword detection in literary corpora
arXiv:cs/0701028 · doi:10.1140/epjb/e2008-00206-x
Abstract
Understanding the complexity of human language requires an appropriate analysis of the statistical distribution of words in texts. We consider the information retrieval problem of detecting and ranking the relevant words of a text by means of statistical information referring to the "spatial" use of the words. Shannon's entropy of information is used as a tool for automatic keyword extraction. By using The Origin of Species by Charles Darwin as a representative text sample, we show the performance of our detector and compare it with another proposals in the literature. The random shuffled text receives special attention as a tool for calibrating the ranking indices.
Published version. 11 pages, 7 figures. SVJour for LaTeX2e
References in corpus (3)
Cited by in corpus (17)
- Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words
- On the origin of long-range correlations in texts
- A complex network approach to stylometry
- Probing the topological properties of complex networks modeling short written texts
- Probing the statistical properties of unknown texts: application to the Voynich Manuscript
- sCAKE: Semantic Connectivity Aware Keyword Extraction
- Towards the quantification of the semantic information encoded in written language
- Complex Network based Supervised Keyword Extractor
- Authorship recognition via fluctuation analysis of network topology and word intermittency
- Comparing intermittency and network measurements of words and their dependency on authorship
- Topic segmentation via community detection in complex networks
- Statistical Patterns in Written Language
- Two halves of a meaningful text are statistically different
- Discriminating word senses with tourist walks in complex networks
- A mathematical model for universal semantics
- Unsupervised extraction of local and global keywords from a single text
- Thermodynamics of Information Retrieval