entropy-aware coding 1large language models 1model compression 1PAC-Bayes bounds 1self-generated training data 1
From the 1 of 3 linked papers with an AI index.
3 papers
cs.LG2026
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
Shikai Qiu, Marc Finzi, Yujia Zheng +2
The paper proposes requential coding, a method where a teacher model selects training samples from the student’s own distribution so that only disagreements need to be encoded, yie…
cs.LG2026
From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
Marc Finzi, Shikai Qiu, Yiding Jiang +3
Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to exist…
cs.LG2025
Compute-Optimal LLMs Provably Generalize Better With Scale
Marc Finzi, Sanyam Kapoor, Diego Granziol +4
Why do larger language models generalize better? To investigate this question, we develop generalization bounds on the pretraining objective of large language models (LLMs) in the…