Showing cs.CLShow all
2 papers · 1 filter
cs.CL2025
Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations
Yize Zhao, Christos Thrampoulidis
We investigate how next-token prediction (NTP) optimization leads language models to extract and organize semantic structure from text. Our analysis, based on a tractable mathemati…
cs.CL2024
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations
Yize Zhao, Tina Behnia, Vala Vakilian +1
Next-token prediction (NTP) over large text corpora has become the go-to paradigm to train large language models. Yet, it remains unclear how NTP influences the mapping of linguist…