5 citations · 8 across the 5 of their papers we have counts for
Showing cs.CLShow all
3 papers · 1 filter
cs.CL2025
Distilling to Hybrid Attention Models via KL-Guided Layer Selection
Yanhong Li, Songlin Yang, Shawn Tan +4
Distilling pretrained softmax attention Transformers into more efficient hybrid architectures that interleave softmax and linear attention layers is a promising approach for improv…
cs.CL2025
PaTH Attention: Position Encoding via Accumulating Householder Transformations
Songlin Yang, Yikang Shen, Kaiyue Wen +5
The attention mechanism is a core primitive in modern large language models (LLMs) and AI more broadly. Since attention by itself is permutation-invariant, position encoding is ess…
cs.CL2023
Simple Hardware-Efficient PCFGs with Independent Left and Right Productions
Wei Liu, Songlin Yang, Yoon Kim +1
Scaling dense PCFGs to thousands of nonterminals via a low-rank parameterization of the rule probability tensor has been shown to be beneficial for unsupervised parsing. However, P…