88 citations · 333 across the 26 of their papers we have counts for
28 papers
RAPTOR: Ridge-Adaptive Logistic Probes
Ziqi Gao, Yaotian Zhu, Qingcheng Zeng +4
Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used opera…
A Compositional Kernel Model for Feature Learning
Feng Ruan, Keli Liu, Michael Jordan
We study a compositional variant of kernel ridge regression in which the predictor is applied to a coordinate-wise reweighting of the inputs. Formulated as a variational problem, t…
Gradient flow in the kernel learning problem
Yang Li, Feng Ruan
This is a sequel to our paper `On the kernel learning problem'. We identify a canonical choice of Riemannian gradient flow, to find the stationary points in the kernel learning pro…
A Variational Analysis of Kernel Learning with Learnable Linear Transformations
Yang Li, Feng Ruan
The classical kernel ridge regression problem aims to find the best fit for the output as a function of the input data , with a fixed choice of regularizatio…
Robust Detection of Watermarks for Large Language Models Under Human Edits
Xiang Li, Feng Ruan, Huiyuan Wang +2
Watermarking has offered an effective approach to distinguishing text generated by large language models (LLMs) from human-written text. However, the pervasive presence of human ed…
On the Limitation of Kernel Dependence Maximization for Feature Selection
Keli Liu, Feng Ruan
A simple and intuitive method for feature selection consists of choosing the feature subset that maximizes a nonparametric measure of dependence between the response and the featur…
On the Uniform Convergence of Subdifferentials in Stochastic Optimization and Learning
Feng Ruan
We investigate the uniform convergence of subdifferential mappings from empirical risk to population risk in nonsmooth, nonconvex stochastic optimization. This question is key to u…
Nonparametric Modern Hopfield Models
Jerry Yao-Chieh Hu, Bo-Yu Chen, Dennis Wu +2
We present a nonparametric interpretation for deep learning compatible modern Hopfield models and utilize this new perspective to debut efficient variants. Our key contribution ste…
A Statistical Framework of Watermarks for Large Language Models: Pivot, Detection Efficiency and Optimal Rules
Xiang Li, Feng Ruan, Huiyuan Wang +2
Since ChatGPT was introduced in November 2022, embedding (nearly) unnoticeable statistical signals into text generated by large language models (LLMs), also known as watermarking,…
A Theory of Feature Learning in Kernel Models
Yunlu Chen, Yang Li, Keli Liu +1
We study feature learning in a compositional variant of kernel ridge regression in which the predictor is applied to a learnable linear transformation of the input. When the respon…
Universality of max-margin classifiers
Andrea Montanari, Feng Ruan, Basil Saeed +1
Maximum margin binary classification is one of the most fundamental algorithms in machine learning, yet the role of featurization maps and the high-dimensional asymptotics of the m…
FastCPH: Efficient Survival Analysis for Neural Networks
Xuelin Yang, Louis Abraham, Sejin Kim +4
The Cox proportional hazards model is a canonical method in survival analysis for prediction of the life expectancy of a patient given clinical or genetic covariates -- it is a lin…
On the Self-Penalization Phenomenon in Feature Selection
Michael I. Jordan, Keli Liu, Feng Ruan
We describe an implicit sparsity-inducing mechanism based on minimization over a family of kernels: \begin{equation*} \min_{β, f}~\widehat{\mathbb{E}}[L(Y, f(β^{1/q} \odot X)] + λ_…
Taming Nonconvexity in Kernel Feature Selection -- Favorable Properties of the Laplace Kernel
Feng Ruan, Keli Liu, Michael I. Jordan
Kernel-based feature selection is an important tool in nonparametric statistics. Despite many practical applications of kernel-based feature selection, there is little statistical…
A Self-Penalizing Objective Function for Scalable Interaction Detection
Keli Liu, Feng Ruan
We tackle the problem of nonparametric variable selection with a focus on discovering interactions between variables. With variables there are possible order- inter…
Bandit Learning in Decentralized Matching Markets
Lydia T. Liu, Feng Ruan, Horia Mania +1
We study two-sided matching markets in which one side of the market (the players) does not have a priori knowledge about its preferences for the other side (the arms) and is requir…
Is Temporal Difference Learning Optimal? An Instance-Dependent Analysis
Koulik Khamaru, Ashwin Pananjady, Feng Ruan +2
We address the problem of policy evaluation in discounted Markov decision processes, and provide instance-dependent guarantees on the -error under a generative model.…
The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
Andrea Montanari, Feng Ruan, Youngtak Sohn +1
Modern machine learning classifiers often exhibit vanishing classification error on the training set. They achieve this by learning nonlinear representations of the inputs that map…
LassoNet: A Neural Network with Feature Sparsity
Ismael Lemhadri, Feng Ruan, Louis Abraham +1
Much work has been done recently to make neural networks more interpretable, and one obvious approach is to arrange for the network to use only a subset of the available features.…
Adapting to Unknown Noise Distribution in Matrix Denoising
Andrea Montanari, Feng Ruan, Jun Yan
We consider the problem of estimating an unknown matrix , from observations where $\bold…
The Right Complexity Measure in Locally Private Estimation: It is not the Fisher Information
John C. Duchi, Feng Ruan
We identify fundamental tradeoffs between statistical utility and privacy under local models of privacy in which data is kept private even from the statistician, providing instance…
A constrained risk inequality for general losses
John C. Duchi, Feng Ruan
We provide a general constrained risk inequality that applies to arbitrary non-decreasing losses, extending a result of Brown and Low [Ann. Stat. 1996]. Given two distributions $P_…
Solving (most) of a set of quadratic equalities: Composite optimization for robust phase retrieval
John C. Duchi, Feng Ruan
We develop procedures, based on minimization of the composition of a convex function and smooth function , for solving random collections of quadratic equal…
Stochastic Methods for Composite and Weakly Convex Optimization Problems
John Duchi, Feng Ruan
We consider minimization of stochastic functionals that are compositions of a (potentially) non-smooth convex function and smooth function and, more generally, stochastic w…