Publications (18)
When is it worthwhile to jackknife? Breaking the quadratic barrier for Z-estimators
Licong Lin, Fangzhou Su, Wenlong Mou +2
Resampling methods are especially well-suited to inference with estimators that provide only "black-box'' access. Jackknife is a form of resampling, widely used for bias correction…
A Statistical Theory of Contrastive Pre-training and Multimodal Generative AI
Kazusato Oko, Licong Lin, Yuhang Cai +1
Multi-modal generative AI systems, such as those combining vision and language, rely on contrastive pre-training to learn representations across different modalities. While their p…
A Statistical Theory of Contrastive Learning via Approximate Sufficient Statistics
Licong Lin, Song Mei
Contrastive learning -- a modern approach to extract useful representations from unlabeled data by training models to distinguish similar samples from dissimilar ones -- has driven…
Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning
Chongyu Fan, Jiancheng Liu, Licong Lin +4
This work studies the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences (e.g., copyrighted or harmful content) while preserving model util…
What causes the test error? Going beyond bias-variance via ANOVA
Licong Lin, Edgar Dobriban
Modern machine learning methods are often overparametrized, allowing adaptation to the data at a fine level. This can seem puzzling; in the worst case, such models do not need to g…
Improved Scaling Laws in Linear Regression via Data Reuse
Licong Lin, Jingfeng Wu, Peter L. Bartlett
Neural scaling laws suggest that the test error of large language models trained online decreases polynomially as the model size and data size increase. However, such scaling can b…
Plug-in Performative Optimization
Licong Lin, Tijana Zrnic
When predictions are performative, the choice of which predictor to deploy influences the distribution of future observations. The overarching goal in learning under performativity…
Statistical Limits of Adaptive Linear Models: Low-Dimensional Estimation and Inference
Licong Lin, Mufang Ying, Suvrojit Ghosh +2
Estimation and inference in statistics pose significant challenges when data are collected adaptively. Even in linear models, the Ordinary Least Squares (OLS) estimator may fail to…
Improving LLM Safety Alignment with Dual-Objective Optimization
Xuandong Zhao, Will Cai, Tianneng Shi +4
Existing training-time safety alignment techniques for large language models (LLMs) remain vulnerable to jailbreak attacks. Direct preference optimization (DPO), a widely deployed…
Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
Ruiqi Zhang, Licong Lin, Yu Bai +1
Large Language Models (LLMs) often memorize sensitive, private, or copyrighted data during pre-training. LLM unlearning aims to eliminate the influence of undesirable data from the…
Mean-field variational inference with the TAP free energy: Geometric and statistical properties in linear models
Michael Celentano, Zhou Fan, Licong Lin +1
We study mean-field variational inference in a Bayesian linear model when the sample size n is comparable to the dimension p. In high dimensions, the common approach of minimizing…
Scaling Laws in Linear Regression: Compute, Parameters, and Data
Licong Lin, Jingfeng Wu, Sham M. Kakade +2
Empirically, large-scale deep learning models often satisfy a neural scaling law: the test error of the trained model improves polynomially as the model size and data size grow. Ho…
Cross-Validated Causal Inference: a Modern Method to Combine Experimental and Observational Data
Xuelin Yang, Licong Lin, Susan Athey +2
We develop new methods to integrate experimental and observational data in causal inference. While randomized controlled trials offer strong internal validity, they are often costl…
Near-optimal multiple testing in Bayesian linear models with finite-sample FDR control
Taejoo Ahn, Licong Lin, Song Mei
In high dimensional variable selection problems, statisticians often seek to design multiple testing procedures that control the False Discovery Rate (FDR), while concurrently iden…
Second-order Information in First-order Optimization Methods
Yuzheng Hu, Licong Lin, Shange Tang
In this paper, we try to uncover the second-order essence of several first-order optimization methods. For Nesterov Accelerated Gradient, we rigorously prove that the algorithm mak…
Minimax Optimal Convergence of Gradient Descent in Logistic Regression via Large and Adaptive Stepsizes
Ruiqi Zhang, Jingfeng Wu, Licong Lin +1
We study (GD) for logistic regression on linearly separable data with stepsizes that adapt to the current risk, scaled by a constant hyperparameter …
Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining
Licong Lin, Yu Bai, Song Mei
Large transformer models pretrained on offline reinforcement learning datasets have demonstrated remarkable in-context reinforcement learning (ICRL) capabilities, where they can ma…
Semi-parametric inference based on adaptively collected data
Licong Lin, Koulik Khamaru, Martin J. Wainwright
Many standard estimators, when applied to adaptively collected data, fail to be asymptotically normal, thereby complicating the construction of confidence intervals. We address thi…