8 papers
Scaling Domain Data Repetition in LLM Pretraining
Jingwei Li, Xinran Gu, Rui Dai +5
As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality doma…
Online Scheduling with a Stochastic Signal
Romain Cosson, Jingwei Li, Alexander Lindermayr +1
Nonclairvoyant scheduling is a fundamental online model in which processing times are initially unknown to the scheduler. Unfortunately, for important objectives such as total comp…
Hallucination is a Consequence of Space-Optimality: A Rate-Distortion Theorem for Membership Testing
Anxin Guo, Jingwei Li
Large language models often hallucinate with high confidence on "random facts" that lack inferable patterns. We formalize the memorization of such facts as a membership testing pro…
Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
Jingwei Li, Xinran Gu, Jingzhao Zhang
A data mixture refers to how different data sources are combined to train large language models, and selecting an effective mixture is crucial for optimal downstream performance. E…
Finite Sample Analyses for Continuous-time Linear Systems: System Identification and Online Control
Hongyi Zhou, Jingwei Li, Jingzhao Zhang
Real world evolves in continuous time but computations are done from finite samples. Therefore, we study algorithms using finite observations in continuous-time linear dynamical sy…
Understanding Nonlinear Implicit Bias via Region Counts in Input Space
Jingwei Li, Jing Xu, Zifan Wang +2
One explanation for the strong generalization ability of neural networks is implicit bias. Yet, the definition and mechanism of implicit bias in non-linear contexts remains little…