4 citations · 12 across the 13 of their papers we have counts for
15 papers
Are Parameter-Efficient Fine-tuning Methods Really Different?
Yikuan Li, Pinyan Lu, Fanghui Liu
Parameter-efficient fine-tuning (PEFT) offers many parameterizations, yet their methodological and functional differences remain unclear. We compare six methods in language and dif…
Compute-Optimal Pretrain--Fine-tune in Ridge Gradient Descent
Alex Buna, Fanghui Liu, Patrick Rebeschini
Pretraining followed by fine-tuning introduces a compute-allocation problem: under a fixed training budget, compute spent improving the upstream objective reduces the compute avail…
Shallow ReLU Networks in -Type and Sobolev Spaces: Approximation and Path-Norm Controlled Generalization
Weizhao Li, Fanghui Liu, Lei Shi
This paper studies approximation by shallow ReLU networks, , together with their generalization behavior under path-norm control. For the -t…
How does feature learning reshape the function space?
João Lobo, Bruno Loureiro, Long Tran-Than +1
Feature learning is widely regarded as the key mechanism distinguishing neural networks from fixed-kernel methods, yet its impact on the induced function space remains poorly under…
Self-Regularized Learning Methods
Max Schölpple, Liu Fanghui, Ingo Steinwart
We introduce a general framework for analyzing learning algorithms based on the notion of self-regularization, which captures implicit complexity control without requiring explicit…
Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy
Tao Li, Weisen Jiang, Fanghui Liu +2
Pre-training followed by fine-tuning is widely adopted among practitioners. The performance can be improved by "model soups"~\cite{wortsman2022model} via exploring various hyperpar…
High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization
Yihang Chen, Fanghui Liu, Taiji Suzuki +1
This paper studies kernel ridge regression in high dimensions under covariate shifts and analyzes the role of importance re-weighting. We first derive the asymptotic expansion of h…
Benign overfitting in Fixed Dimension via Physics-Informed Learning with Smooth Inductive Bias
Honam Wong, Wendao Wu, Fanghui Liu +1
Recent advances in machine learning have inspired a surge of research into reconstructing specific quantities of interest from measurements that comply with certain physical laws.…
Revisiting Character-level Adversarial Attacks for Language Models
Elias Abad Rocamora, Yongtao Wu, Fanghui Liu +2
Adversarial attacks in Natural Language Processing apply perturbations in the character or token levels. Token-level attacks, gaining prominence for their use of gradient-based met…
Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks
Fanghui Liu, Leello Dadi, Volkan Cevher
Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks as the curse of dimensionality (CoD) cannot be evad…
Robust NAS under adversarial training: benchmark, theory, and beyond
Yongtao Wu, Fanghui Liu, Carl-Johann Simon-Gabriel +2
Recent developments in neural architecture search (NAS) emphasize the significance of considering robust architectures against malicious data. However, there is a notable absence o…
Generalization of Scaled Deep ResNets in the Mean-Field Regime
Yihang Chen, Fanghui Liu, Yiping Lu +2
Despite the widespread empirical success of ResNet, the generalization properties of deep ResNet are rarely explored beyond the lazy training regime. In this work, we investigate \…
Efficient local linearity regularization to overcome catastrophic overfitting
Elias Abad Rocamora, Fanghui Liu, Grigorios G. Chrysos +2
Catastrophic overfitting (CO) in single-step adversarial training (AT) results in abrupt drops in the adversarial test accuracy (even down to 0%). For models trained with multi-ste…
On the Convergence of Encoder-only Shallow Transformers
Yongtao Wu, Fanghui Liu, Grigorios G Chrysos +1
In this paper, we aim to build the global convergence theory of encoder-only shallow Transformers under a realistic setting from the perspective of architectures, initialization, a…
Initialization Matters: Privacy-Utility Analysis of Overparameterized Neural Networks
Jiayuan Ye, Zhenyu Zhu, Fanghui Liu +2
We analytically investigate how over-parameterization of models in randomized machine learning algorithms impacts the information leakage about their training data. Specifically, w…