Generalization Bounds for Transformer-Based Next-Token Prediction in a Language Model

Insung Kong, Niklas Dexheimer, Johannes Schmidt-Hieber

A refined statistical understanding of LLM pre-training requires the analysis of the transformer architecture for data distributions that encapsulate key characteristics of text da…