Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
Chuanyang Zheng, Jiankai Sun, Yihang Gao +11
The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches t…
cs.LG2025
Swapped Logit Distillation via Bi-level Teacher Alignment
Stephen Ekaputra Limantoro, Jhe-Hao Lin, Chih-Yu Wang +4
Knowledge distillation (KD) compresses the network capacity by transferring knowledge from a large (teacher) network to a smaller one (student). It has been mainstream that the tea…