Universal and non-universal text statistics: Clustering coefficient for language identification

Abstract

In this work we analyze statistical properties of 91 relatively small texts in 7 different languages (Spanish, English, French, German, Turkish, Russian, Icelandic) as well as texts with randomly inserted spaces. Despite the size (around 11260 different words), the well known universal statistical laws -- namely Zipf and Herdan-Heap's laws -- are confirmed, and are in close agreement with results obtained elsewhere. We also construct a word co-occurrence network of each text. While the degree distribution is again universal, we note that the distribution of Clustering Coefficients, which depend strongly on the local structure of networks, can be used to differentiate between languages, as well as to distinguish natural languages from random texts.

arXiv comment15 pages, 6 figures

References in corpus (9)

• Power-law distributions in empirical data
• Large-scale analysis of Zipf's law in English texts
• Mean-field theory for clustering coefficients in Barabasi-Albert networks
• Text authorship identified using the dynamics of word co-occurrence networks
• Statistical laws in linguistics
• Probing the statistical properties of unknown texts: application to the Voynich Manuscript
• Measures of lexical distance between languages
• Tail universalities in rank distributions as an algebraic problem: the beta-like function
• Statistical Patterns in Written Language