Information-Theoretic Analysis of Next-Token Prediction under Markovian Data

Masoud Kavian, Abdellatif Zaidi, Milad Sefidgaran

We develop an information-theoretic framework for generalization in next-token prediction under temporally dependent data. We consider independent trajectories generated by finite-…