10 citations · 32 across the 22 of their papers we have counts for
22 papers
Tail-Likelihood Reinforcement Learning
Shrinivas Ramasubramanian, Daman Arora, Fahim Tajwar +11
Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward whi…
Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding
Fengxiang Bie, Yuqing Jian, Yifan Yu +7
Speculative decoding is critical for accelerating LLM inference. However, the speedup is fragile: drafters are typically trained against a narrow distribution for a single target m…
Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
Zunhai Su, Bohan Sun, Xialie Zhuang +8
We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediate…
Linear RNN Scaling Laws: When Longer Sequences Beat More Sequences
Ziyan Chen, Zhongzhu Zhou, Peilin Liu +1
Empirical scaling laws for autoregressive language models relate prediction loss to model size, data size, and optimization compute, but their theoretical origin is still poorly un…
Sketched Linear Contrastive Learning: Approximation, Optimization, and Statistical Scaling
Ziyan Chen, Zhongzhu Zhou, Ding-Xuan Zhou
Scaling laws describe how learning performance varies with model size, data size, and compute. While recent theoretical work has established scaling laws for sketched linear regres…
Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation
Zhongzhu Zhou, Qingyang Wu, Junxiong Wang +4
Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retainin…