43 citations · 55 across the 20 of their papers we have counts for
21 papers
On the Recoverability of Private Information Unlearning in Large Language Models
Shicheng Hu, Runzhi Tian, Ziqiao Wang +1
Large language models (LLMs) can memorize sensitive information, raising serious privacy concerns. Machine unlearning offers a potential solution to remove such information, but it…
Fundamental Trade-Offs in Multi-Bit Watermarking of Stochastic Processes
Haiyun He, Yepeng Liu, Zhuoer Shen +3
We study multi-bit watermarking for data generated by stochastic processes, where a hidden message is embedded during sampling and must be decodable by an authorized detector that…
On the Blessing of Pre-training in Weak-to-Strong Generalization
Wei Yao, Wang Zhaoyang, Gengze Xu +5
The paradigm of Weak-to-Strong Generalization (W2SG) suggests that a pre-trained strong model can surpass its weak supervisor, yet the decisive role of pre-training remains theoret…
Generalization in Federated Learning: A Conditional Mutual Information Framework
Ziqiao Wang, Cheng Long, Yongyi Mao
Federated learning (FL) is a widely adopted privacy-preserving distributed learning framework, yet its generalization performance remains less explored compared to centralized lear…
Revisiting Weak-to-Strong Generalization in Theory and Practice: Reverse KL vs. Forward KL
Wei Yao, Wenkai Yang, Ziqiao Wang +2
As large language models advance toward superhuman performance, ensuring their alignment with human values and abilities grows increasingly complex. Weak-to-strong generalization o…
The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration
Wei Yao, Wenkai Yang, Gengze Xu +3
Weak-to-strong generalization, where weakly supervised strong models outperform their weaker teachers, offers a promising approach to aligning superhuman models with human values.…
Distributional Information Embedding: A Framework for Multi-bit Watermarking
Haiyun He, Yepeng Liu, Ziqiao Wang +2
This paper introduces a novel problem, distributional information embedding, motivated by the practical demands of multi-bit watermarking for large language models (LLMs). Unlike t…
LH-Mix: Local Hierarchy Correlation Guided Mixup over Hierarchical Prompt Tuning
Fanshuang Kong, Richong Zhang, Ziqiao Wang
Hierarchical text classification (HTC) aims to assign one or more labels in the hierarchy for each text. Many methods represent this structure as a global hierarchy, leading to red…
MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging
Fanshuang Kong, Richong Zhang, Zhijie Nie +4
Model merging offers a scalable alternative to multi-task learning but often yields suboptimal performance on classification tasks. We attribute this degradation to a geometric mis…
Generalization Bounds via Conditional -Information
Ziqiao Wang, Yongyi Mao
In this work, we introduce novel information-theoretic generalization bounds using the conditional -information framework, an extension of the traditional conditional mutual inf…
Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach
Haiyun He, Yepeng Liu, Ziqiao Wang +2
Watermarking has emerged as a crucial method to distinguish AI-generated text from human-created text. Current watermarking approaches often lack formal optimality guarantees or ad…
Activated Parameter Locating via Causal Intervention for Model Merging
Fanshuang Kong, Richong Zhang, Ziqiao Wang
Model merging combines multiple homologous models into one model, achieving convincing generalization without the necessity of additional training. A key challenge in this problem…
Cross-Modal and Uni-Modal Soft-Label Alignment for Image-Text Retrieval
Hailang Huang, Zhijie Nie, Ziqiao Wang +1
Current image-text retrieval methods have demonstrated impressive performance in recent years. However, they still face two problems: the inter-modal matching missing problem and t…
On -Divergence Principled Domain Adaptation: An Improved Framework
Ziqiao Wang, Yongyi Mao
Unsupervised domain adaptation (UDA) plays a crucial role in addressing distribution shifts in machine learning. In this work, we improve the theoretical foundations of UDA propose…
Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Bounds
Ziqiao Wang, Yongyi Mao
We present new information-theoretic generalization guarantees through the a novel construction of the "neighboring-hypothesis" matrix and a new family of stability notions termed…
Over-training with Mixup May Hurt Generalization
Zixuan Liu, Ziqiao Wang, Hongyu Guo +1
Mixup, which creates synthetic training instances by linearly interpolating random sample pairs, is a simple and yet effective regularization technique to boost the performance of…
Tighter Information-Theoretic Generalization Bounds from Supersamples
Ziqiao Wang, Yongyi Mao
In this work, we present a variety of novel information-theoretic generalization bounds for learning algorithms, from the supersample setting of Steinke & Zakynthinou (2020)-the se…
Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States
Ziqiao Wang, Yongyi Mao
Stochastic differential equations (SDEs) have been shown recently to characterize well the dynamics of training machine learning models with SGD. When the generalization error of t…
Information-Theoretic Analysis of Unsupervised Domain Adaptation
Ziqiao Wang, Yongyi Mao
This paper uses information-theoretic tools to analyze the generalization error in unsupervised domain adaptation (UDA). We present novel upper bounds for two notions of generaliza…
On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications
Ziqiao Wang, Yongyi Mao
This paper follows up on a recent work of Neu et al. (2021) and presents some new information-theoretic upper bounds for the generalization error of machine learning models, such a…
On SkipGram Word Embedding Models with Negative Sampling: Unified Framework and Impact of Noise Distributions
Dezhi Liu, Richong Zhang, Ziqiao Wang
SkipGram word embedding models with negative sampling, or SGN in short, is an elegant family of word embedding models. In this paper, we formulate a framework for word embedding, r…