24 citations · 24 across the 5 of their papers we have counts for
5 papers
Scaling Domain Data Repetition in LLM Pretraining
Jingwei Li, Xinran Gu, Rui Dai +5
As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality doma…
Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
Jingwei Li, Xinran Gu, Jingzhao Zhang
A data mixture refers to how different data sources are combined to train large language models, and selecting an effective mixture is crucial for optimal downstream performance. E…
Data Mixing Can Induce Phase Transitions in Knowledge Acquisition
Xinran Gu, Kaifeng Lyu, Jiazheng Li +1
Large Language Models (LLMs) are typically trained on data mixtures: most data come from web scrapes, while a small portion is curated from high-quality sources with dense domain-s…
A Quadratic Synchronization Rule for Distributed Deep Learning
Xinran Gu, Kaifeng Lyu, Sanjeev Arora +2
In distributed deep learning with data parallelism, synchronizing gradients at each training step can cause a huge communication overhead, especially when many nodes work together…
Fast Federated Learning in the Presence of Arbitrary Device Unavailability
Xinran Gu, Kaixuan Huang, Jingzhao Zhang +1
Federated Learning (FL) coordinates with numerous heterogeneous devices to collaboratively train a shared model while preserving user privacy. Despite its multiple advantages, FL f…