5 citations · 8 across the 5 of their papers we have counts for
Showing 2025Show all
3 papers · 1 filter
cs.CL2025
Distilling to Hybrid Attention Models via KL-Guided Layer Selection
Yanhong Li, Songlin Yang, Shawn Tan +4
Distilling pretrained softmax attention Transformers into more efficient hybrid architectures that interleave softmax and linear attention layers is a promising approach for improv…
cs.LG2025
Log-Linear Attention
Han Guo, Songlin Yang, Tarushii Goel +3
The attention mechanism in Transformers is an important primitive for accurate and scalable sequence modeling. Its quadratic-compute and linear-memory complexity however remain sig…
cs.CL2025
PaTH Attention: Position Encoding via Accumulating Householder Transformations
Songlin Yang, Yikang Shen, Kaiyue Wen +5
The attention mechanism is a core primitive in modern large language models (LLMs) and AI more broadly. Since attention by itself is permutation-invariant, position encoding is ess…