Training LLMs over Neurally Compressed Text

Brian Lester, Jaehoon Lee, Alex Alemi +4

In this paper, we explore the idea of training large language models (LLMs) over highly compressed text. While standard subword tokenizers compress text by a small factor, neural t…