11 citations · 11 across the 9 of their papers we have counts for
4 papers · 1 filter
SPLA: Block Sparse Plus Linear Attention for Long Context Modeling
Bailin Wang, Dan Friedman, Tao Lei +1
Block-wise sparse attention offers significant efficiency gains for long-context modeling, yet existing methods often suffer from low selection fidelity and cumulative contextual l…
IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining
Yixiao Li, Xianzhi Du, Ajay Jaiswal +4
Recent advancements in large language models have intensified the need for efficient and deployable models within limited inference budgets. Structured pruning pipelines have shown…
Instruction-Following Pruning for Large Language Models
Bairu Hou, Qibin Chen, Jianyu Wang +6
With the rapid scaling of large language models (LLMs), structured pruning has become a widely used technique to learn efficient, smaller models from larger ones, delivering superi…
Learning to Skip for Language Modeling
Dewen Zeng, Nan Du, Tao Wang +4
Overparameterized large-scale language models have impressive generalization performance of in-context few-shot learning. However, most language models allocate the same amount of…