5 papers
SiameseNorm: Breaking the Barrier to Reconciling Pre/Post-Norm
Tianyu Li, Dongchen Han, Zixuan Cao +7
The long-standing tension between Pre- and Post-Norm remains an open problem in Transformer architecture, reflecting a fundamental trade-off between training stability and represen…
ViT: Unlocking Test-Time Training in Vision
Dongchen Han, Yining Li, Tianyu Li +6
Test-Time Training (TTT) has recently emerged as a promising direction for efficient sequence modeling. TTT reformulates attention operation as an online learning problem, construc…
Vision Transformers are Circulant Attention Learners
Dongchen Han, Tianyu Li, Ziyi Wang +1
The self-attention mechanism has been a key factor in the advancement of vision Transformers. However, its quadratic complexity imposes a heavy computational burden in high-resolut…
Step by Step Network
Dongchen Han, Tianzhu Ye, Zhuofan Xia +4
Scaling up network depth is a fundamental pursuit in neural architecture design, as theory suggests that deeper models offer exponentially greater capability. Benefiting from the r…
Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials
Yifan Pu, Jixuan Ying, Qixiu Li +7
Vision Transformers (ViTs) have become a universal backbone for both image recognition and image generation. Yet their Multi-Head Self-Attention (MHSA) layer still performs a quadr…