works on
efficient training 1large language models 1residual stream expansion 1sparse computation 1transformer architectures 1

From the 1 of 2 linked papers with an AI index.

collaborators
Showing cs.CLShow all

1 paper · 1 filter

cs.CL2026

NITP: Next Implicit Token Prediction for LLM Pre-training

Xiangdong Zhang, Debing Zhang, Shaofeng Zhang +3

Standard next-token prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse one-hot supervision leaves the l…