1 citations · 2 across the 6 of their papers we have counts for
Showing 2025 · cs.CLShow all
2 papers · 2 filters
cs.CL2025
Distilling to Hybrid Attention Models via KL-Guided Layer Selection
Yanhong Li, Songlin Yang, Shawn Tan +4
Distilling pretrained softmax attention Transformers into more efficient hybrid architectures that interleave softmax and linear attention layers is a promising approach for improv…
cs.CL2025
PaTH Attention: Position Encoding via Accumulating Householder Transformations
Songlin Yang, Yikang Shen, Kaiyue Wen +5
The attention mechanism is a core primitive in modern large language models (LLMs) and AI more broadly. Since attention by itself is permutation-invariant, position encoding is ess…