7 papers · 1 filter
Learning Style, Forgetting Semantics: A Case Study of SFT and RFT on Classification Tasks
Haodong Liang, Yanhao Jin, Krishnakumar Balasubramanian +1
Why does supervised fine-tuning (SFT) lead to more forgetting than reinforcement fine-tuning (RFT), even when all teacher demonstrations are semantically correct? We study this que…
Finite-Particle Convergence Rates for Conservative and Non-Conservative Drifting Models
Krishnakumar Balasubramanian
We analyze finite-particle drifting models for one-step generative modeling. For a conservative velocity given by the difference of the kernel-smoothed data and model scores, a joi…
Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attention
Krishnakumar Balasubramanian
Unbiased stochastic gradients can match the full gradient in expectation while changing finite-step stability. We study this effect in one-layer linear self-attention for in-contex…
Differentially Private Two-Stage Gradient Descent for Instrumental Variable Regression
Haodong Liang, Yanhao Jin, Krishnakumar Balasubramanian +1
We study instrumental variable regression (IVaR) under differential privacy constraints. Classical IVaR methods (like two-stage least squares regression) rely on solving moment equ…
Finite-Dimensional Gaussian Approximation for Deep Neural Networks: Universality in Random Weights
Krishnakumar Balasubramanian, Nathan Ross
We study the Finite-Dimensional Distributions (FDDs) of deep neural networks with randomly initialized weights that have finite-order moments. Specifically, we establish Gaussian a…
In-context Learning for Mixture of Linear Regressions: Existence, Generalization and Training Dynamics
Yanhao Jin, Krishnakumar Balasubramanian, Lifeng Lai
We investigate the in-context learning capabilities of transformers for the -dimensional mixture of linear regression model, providing theoretical insights into their existence,…
Transformers Handle Endogeneity in In-Context Linear Regression
Haodong Liang, Krishnakumar Balasubramanian, Lifeng Lai
We explore the capability of transformers to address endogeneity in in-context linear regression. Our main finding is that transformers inherently possess a mechanism to handle end…