22 papers
S-SPPO: Semantic-Calibrated Self-Play Preference Optimization
Xiwen Chen, Wenhui Zhu, Jingjing Wang +13
Aligning Large Language Models (LLMs) with human preferences is often formulated via Direct Preference Optimization (DPO). However, the standard Bradley-Terry instantiation of DPO…
A Mechanistic Study of Tabular Foundation Models
Marin Biloš, James T. Wilson, Anderson Schneider +1
Tabular foundation models with different architectures converge in accuracy across a range of classification and regression tasks. This raises questions a leaderboard cannot answer…
Cubit: Token Mixer with Kernel Ridge Regression
Chuanyang Zheng, Jiankai Sun, Yihang Gao +6
Since its introduction in 2017, the Transformer has become one of the most widely adopted architectures in modern deep learning. Despite extensive efforts to improve positional enc…
AlphaLab: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMs
Brendan R. Hogan, Xiwen Chen, James T. Wilson +5
We present AlphaLab, an autonomous research harness that leverages frontier LLM agentic capabilities to automate the full experimental cycle in quantitative, computation-intensive…
Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
Kevin Rojas, Jiahe Lin, Kashif Rasul +4
Diffusion language models (DLMs) enable parallel, order-agnostic generation with iterative refinement, offering a flexible alternative to autoregressive large language models (LLMs…
GeoNorm: Unify Pre-Norm and Post-Norm with Geodesic Optimization
Chuanyang Zheng, Jiankai Sun, Yihang Gao +11
The placement of normalization layers, specifically Pre-Norm and Post-Norm, remains an open question in Transformer architecture design. In this work, we rethink these approaches t…