6 papers · 1 filter
Learning to Extrapolate in Depth with Looped Transformers
Huitao Yang, Jianhao Huang, Zixuan Wang +1
Looped transformers reuse a single block across iterations, raising a natural question: how far can a block trained at one depth be safely extrapolated to greater depths? We study…
How Transformers Learn to Plan via Multi-Token Prediction
Jianhao Huang, Zhanpeng Zhou, Renqiu Xia +3
While next-token prediction (NTP) has been the standard objective for training language models, it often struggles to capture global structure in reasoning tasks. Multi-token predi…
Beyond What Seems Necessary: Hidden Gains from Scaling Training-Time Reasoning Length under Outcome Supervision
Yihao Xue, Allan Zhang, Jianhao Huang +2
Training LLMs to think and reason for longer has become a key ingredient in building state-of-the-art models that can solve complex problems previously out of reach. Recent efforts…
Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from -Parity
Jianhao Huang, Baharan Mirzasoleiman
Masked Diffusion Language Models have recently emerged as a powerful generative paradigm, yet their generalization properties remain understudied compared to their auto-regressive…
SL-ACC: A Communication-Efficient Split Learning Framework with Adaptive Channel-wise Compression
Zehang Lin, Zheng Lin, Miao Yang +7
The increasing complexity of neural networks poses a significant barrier to the deployment of distributed machine learning (ML) on resource-constrained devices, such as federated l…
Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thought
Jianhao Huang, Zixuan Wang, Jason D. Lee
Chain of Thought (CoT) prompting has been shown to significantly improve the performance of large language models (LLMs), particularly in arithmetic and reasoning tasks, by instruc…