5 papers · 1 filter
An Information-Theoretic Perspective on LLM Tokenizers
Mete Erdogan, Abhiram Gorle, Shubham Chandak +2
Large language model (LLM) tokenizers act as structured compressors: by mapping text to discrete token sequences, they determine token count (and thus compute and context usage) an…
The Performance of Compression-Based Denoisers
Dan Song, Ayfer Özgür, Tsachy Weissman
We consider a denoiser that reconstructs a stationary ergodic source by lossily compressing samples of the source observed through a memoryless noisy channel. Prior work on compres…
Information-computation trade-offs in non-linear transforms
Connor Ding, Abhiram Rao Gorle, Jiwon Jeong +2
In this work, we explore the interplay between information and computation in non-linear transform-based compression for broad classes of modern information-processing tasks. We fi…
The LZ78 Source
Naomi Sagan, Amir Dembo, Matthew Ho +1
We study a family of processes generated according to sequential probability assignments induced by the LZ78 universal compressor. We characterize entropic and distributional prope…
A Family of LZ78-based Universal Sequential Probability Assignments
Naomi Sagan, Tsachy Weissman
We propose and study a family of universal sequential probability assignments on individual sequences, based on the incremental parsing procedure of the Lempel-Ziv (LZ78) compressi…