39 citations · 48 across the 9 of their papers we have counts for
3 papers · 1 filter
The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws
Tian Jin, Ahmed Imtiaz Humayun, Utku Evci +4
Pruning eliminates unnecessary parameters in neural networks; it offers a promising solution to the growing computational demands of large language models (LLMs). While many focus…
Striped Attention: Faster Ring Attention for Causal Transformers
William Brandon, Aniruddha Nrusimha, Kevin Qian +4
To help address the growing demand for ever-longer sequence lengths in transformer models, Liu et al. recently proposed Ring Attention, an exact attention algorithm capable of over…
Pruning's Effect on Generalization Through the Lens of Training and Regularization
Tian Jin, Michael Carbin, Daniel M. Roy +2
Practitioners frequently observe that pruning improves model generalization. A long-standing hypothesis based on bias-variance trade-off attributes this generalization improvement…