8 papers
Plan, Verify and Fill: A Structured Parallel Decoding Approach for Diffusion Language Models
Miao Li, Hanyang Jiang, Sikai Cheng +6
Diffusion Language Models (DLMs) present a promising non-sequential paradigm for text generation, distinct from standard autoregressive (AR) approaches. However, current decoding s…
LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws
Xu Ouyang, Deyi Liu, Yuhang Cai +5
Existing scaling laws for Large Language Models (LLMs), predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and q…
A Statistical Theory of Contrastive Pre-training and Multimodal Generative AI
Kazusato Oko, Licong Lin, Yuhang Cai +1
Multi-modal generative AI systems, such as those combining vision and language, rely on contrastive pre-training to learn representations across different modalities. While their p…
GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
Chen Zheng, Yuhang Cai, Deyi Liu +7
Modern large language models leverage Mixture-of-Experts (MoE) architectures for efficient scaling, but face a critical challenge: functionally similar experts are often selected s…
Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks
Yuhang Cai, Kangjie Zhou, Jingfeng Wu +3
We establish the asymptotic implicit bias of gradient descent (GD) for generic non-homogeneous deep networks under exponential loss. Specifically, we characterize three key propert…
Non-Euclidean dual gradient ascent for entropically regularized linear and semidefinite programming
Yuhang Cai, Michael Lindsey
We present an optimization framework that exhibits dimension-independent convergence on a broad class of semidefinite programs (SDPs). Our approach first regularizes the primal pro…