4 papers · 1 filter
Interactive-Policy Distillation with Bidirectional Propose-and-Verify
Shutong Wu, Xiwen Chen, Brendan Rappazzo +4
On-policy distillation (OPD) trains a student model on its self-generated trajectories with dense token-level teacher feedback. However, naive OPD may suffer from teacher unanchori…
A Mechanistic Study of Tabular Foundation Models
Marin Biloš, James T. Wilson, Anderson Schneider +1
Tabular foundation models with different architectures converge in accuracy across a range of classification and regression tasks. This raises questions a leaderboard cannot answer…
Cubit: Token Mixer with Kernel Ridge Regression
Chuanyang Zheng, Jiankai Sun, Yihang Gao +6
Since its introduction in 2017, the Transformer has become one of the most widely adopted architectures in modern deep learning. Despite extensive efforts to improve positional enc…
GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
Chuanyang Zheng, Jiankai Sun, Yihang Gao +11
The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches t…