3 papers
cs.CL2026
Short-Context Dominance: How Much Local Context Natural Language Actually Needs?
Vala Vakilian, Zimeng Wang, Ankit Singh Rawat +1
We investigate the short-context dominance hypothesis: that for most sequences, a small local prefix suffices to predict their next tokens. Using large language models as statistic…
cs.LG2025
DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models
Wenlong Deng, Yize Zhao, Vala Vakilian +3
Storing open-source fine-tuned models separately introduces redundancy and increases response times in applications utilizing multiple models. Delta-parameter pruning (DPP), partic…
cs.CL2025
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations
Yize Zhao, Tina Behnia, Vala Vakilian +1
Next-token prediction (NTP) over large text corpora has become the go-to paradigm to train large language models. Yet, it remains unclear how NTP influences the mapping of linguist…