14 citations · 31 across the 27 of their papers we have counts for
4 papers · 2 filters
Intern-S2-Preview: Scientific Agentic Foundation Model
Lei Bai, Jiaqi Cao, Chiyu Chen +121
Scientific discovery increasingly requires AI systems that can reason over scientific evidence of heterogeneous modalities, interact with scientific tools and environments, and sus…
Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
Yechen Zhang, Shuhao Xing, Junhao Huang +5
Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improve g…
What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs
Runyu Peng, Ruixiao Li, Mingshu Chen +5
Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as attention sinks. Causal large language models reliably form one at position ze…
Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
Runyu Peng, Yunhua Zhou, Demin Song +4
In large language models built upon the Transformer architecture, recent studies have shown that inter-head interaction can enhance attention performance. Motivated by this, we pro…