Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models

Wenqi Zeng, Chen Feng, YuAng Chen

Hybrid language models interleave linear-attention layers with a small number of softmax-attention layers. Prior work largely attributes in-context retrieval to softmax attention,…