Showing cs.CLShow all
3 papers · 1 filter
cs.CL2026
Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Head
Zizhuo Fu, Runsheng Wang, Meng Li
Scaling attention parameters can improve language model quality, but retaining full token histories makes additional heads costly at long contexts. Furthermore, since attention ret…
cs.CL2026
Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
Zizhuo Fu, Wenxuan Zeng, Runsheng Wang +1
Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this iss…
cs.CL2025
PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Models
Shi Qiu, Shaoyang Guo, Zhuo-Yang Song +51
Current benchmarks for evaluating the reasoning capabilities of Large Language Models (LLMs) face significant limitations: task oversimplification, data contamination, and flawed e…