Large language models and the entropy of English

Colin Scheibner, Lindsay M. Smith, William Bialek

We use large language models (LLMs) to uncover long-ranged structure in English texts from a variety of sources. The conditional entropy or code length in many cases continues to d…