4 citations · 4 across the 3 of their papers we have counts for
3 papers · 1 filter
Ranking and Partitioning of Attention Heads for Sparse and Continual Reinforcement Learning Fine-tuning
Ruizhe Shi, Zaiwen Yang, Qiubo Wang +2
We study sparse and continual reinforcement learning fine-tuning of language models through attention-head ranking and partitioning. For sparse learning, we rank attention heads by…
Gating is Weighting: Understanding Gated Linear Attention through In-context Learning
Yingcong Li, Davoud Ataee Tarzanagh, Ankit Singh Rawat +2
Linear attention methods offer a compelling alternative to softmax attention due to their efficiency in recurrent decoding. Recent research has focused on enhancing standard linear…
Towards Sample-efficient Overparameterized Meta-learning
Yue Sun, Adhyyan Narang, Halil Ibrahim Gulluk +2
An overarching goal in machine learning is to build a generalizable model with few samples. To this end, overparameterization has been the subject of immense interest to explain th…