43 citations · 55 across the 20 of their papers we have counts for
9 papers · 1 filter
On the Recoverability of Private Information Unlearning in Large Language Models
Shicheng Hu, Runzhi Tian, Ziqiao Wang +1
Large language models (LLMs) can memorize sensitive information, raising serious privacy concerns. Machine unlearning offers a potential solution to remove such information, but it…
On the Blessing of Pre-training in Weak-to-Strong Generalization
Wei Yao, Wang Zhaoyang, Gengze Xu +5
The paradigm of Weak-to-Strong Generalization (W2SG) suggests that a pre-trained strong model can surpass its weak supervisor, yet the decisive role of pre-training remains theoret…
Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL
Wei Yao, Wenkai Yang, Ziqiao Wang +2
As large language models advance toward superhuman performance, ensuring their alignment with human values and abilities grows increasingly complex. Weak-to-strong generalization o…
The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration
Wei Yao, Wenkai Yang, Gengze Xu +3
Weak-to-strong generalization, where weakly supervised strong models outperform their weaker teachers, offers a promising approach to aligning superhuman models with human values.…
MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging
Fanshuang Kong, Richong Zhang, Zhijie Nie +4
Model merging offers a scalable alternative to multi-task learning but often yields suboptimal performance on classification tasks. We attribute this degradation to a geometric mis…
Over-training with Mixup May Hurt Generalization
Zixuan Liu, Ziqiao Wang, Hongyu Guo +1
Mixup, which creates synthetic training instances by linearly interpolating random sample pairs, is a simple and yet effective regularization technique to boost the performance of…
Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States
Ziqiao Wang, Yongyi Mao
Stochastic differential equations (SDEs) have been shown recently to characterize well the dynamics of training machine learning models with SGD. When the generalization error of t…
Information-Theoretic Analysis of Unsupervised Domain Adaptation
Ziqiao Wang, Yongyi Mao
This paper uses information-theoretic tools to analyze the generalization error in unsupervised domain adaptation (UDA). We present novel upper bounds for two notions of generaliza…
On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications
Ziqiao Wang, Yongyi Mao
This paper follows up on a recent work of Neu et al. (2021) and presents some new information-theoretic upper bounds for the generalization error of machine learning models, such a…