Showing cs.LGShow all
3 papers · 1 filter
cs.LG2026
Cubit: Token Mixer with Kernel Ridge Regression
Chuanyang Zheng, Jiankai Sun, Yihang Gao +6
Since its introduction in 2017, the Transformer has become one of the most widely adopted architectures in modern deep learning. Despite extensive efforts to improve positional enc…
cs.LG2026
GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
Chuanyang Zheng, Jiankai Sun, Yihang Gao +11
The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches t…
cs.LG2025
Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
Peihao Wang, Ruisi Cai, Yuehao Wang +4
Structured State Space Models (SSMs) have emerged as alternatives to transformers. While SSMs are often regarded as effective in capturing long-sequence dependencies, we rigorously…