1 citations · 1 across the 10 of their papers we have counts for
8 papers · 1 filter
Bridging Compute- and Data-Optimal Pretraining
Tian Qin, Kimia Hamidieh, David Alvarez-Melis
Classical compute-optimal scaling laws assume an unbounded supply of fresh pretraining data, yet pretraining is increasingly entering a regime in which compute grows faster than th…
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
Rachit Bansal, Clara Mohri, Tian Qin +2
The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from…
Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power
Yuzhu Chen, Tian Qin, Xinmei Tian +2
Equivariant neural networks encode the intrinsic symmetry of data as an inductive bias, which has achieved impressive performance in wide domains. However, the understanding to the…
To Backtrack or Not to Backtrack: When Sequential Search Limits Model Reasoning
Tian Qin, David Alvarez-Melis, Samy Jelassi +1
Recent advancements in large language models (LLMs) have significantly improved their reasoning abilities, particularly through techniques involving search and backtracking. Backtr…
Random Scaling of Emergent Capabilities
Rosie Zhao, Tian Qin, David Alvarez-Melis +2
Language models famously improve under a smooth scaling law, but some specific capabilities exhibit sudden breakthroughs in performance. Advocates of "emergence" view these capabil…
Sometimes I am a Tree: Data Drives Unstable Hierarchical Generalization
Tian Qin, Naomi Saphra, David Alvarez-Melis
Early in training, LMs can behave like n-gram models, but eventually they often learn tree-based syntactic rules and generalize hierarchically out of distribution (OOD). We study t…
A Label is Worth a Thousand Images in Dataset Distillation
Tian Qin, Zhiwei Deng, David Alvarez-Melis
Data is a crucial factor in the performance of machine learning models, a principle that dataset distillation methods exploit by compressing training datasets in…
Distributional Dataset Distillation with Subtask Decomposition
Tian Qin, Zhiwei Deng, David Alvarez-Melis
What does a neural network learn when training from a task-specific dataset? Synthesizing this knowledge is the central idea behind Dataset Distillation, which recent work has show…