6 papers
The Geometry of Statistical Feature Learning in Mean-Field Langevin Dynamics
Zong Shang, Tomoya Wakayama, Guillaume Lecué +1
We introduce a geometric formulation of statistical feature learning for supervised regression. Feature learning is defined through a base--fiber decomposition: the base is the fea…
Towards a Unified Analysis of Neural Networks in Nonparametric Instrumental Variable Regression: Optimization and Generalization
Zonghao Chen, Atsushi Nitanda, Arthur Gretton +1
We establish the first global convergence result of neural networks for two stage least squares (2SLS) approach in nonparametric instrumental variable regression (NPIV). This is ac…
Provable In-Context Vector Arithmetic via Retrieving Task Concepts
Dake Bu, Wei Huang, Andi Han +4
In-context learning (ICL) has garnered significant attention for its ability to grasp functions/tasks from demonstrations. Recent studies suggest the presence of a latent task/func…
Direct Distributional Optimization for Provable Alignment of Diffusion Models
Ryotaro Kawata, Kazusato Oko, Atsushi Nitanda +1
We introduce a novel alignment method for diffusion models from distribution optimization perspectives while providing rigorous convergence guarantees. We first formulate the probl…
Propagation of Chaos for Mean-Field Langevin Dynamics and its Application to Model Ensemble
Atsushi Nitanda, Anzelle Lee, Damian Tan Xing Kai +2
Mean-field Langevin dynamics (MFLD) is an optimization method derived by taking the mean-field limit of noisy gradient descent for two-layer neural networks in the mean-field regim…
Gradient-Based Feature Learning under Structured Data
Alireza Mousavi-Hosseini, Denny Wu, Taiji Suzuki +1
Recent works have demonstrated that the sample complexity of gradient-based learning of single index models, i.e. functions that depend on a 1-dimensional projection of the input d…