Physics in Next-token Prediction

Hongjun An, Yiliang Song, Xuelong Li

We discovered the underlying physics in Next-token Prediction (NTP). We identified the law of information conservation within NTP and proposed the First Law of Information Capacity…