12 papers
Fundamental Trade-Offs in Multi-Bit Watermarking of Stochastic Processes
Haiyun He, Yepeng Liu, Zhuoer Shen +3
We study multi-bit watermarking for data generated by stochastic processes, where a hidden message is embedded during sampling and must be decodable by an authorized detector that…
On the Blessing of Pre-training in Weak-to-Strong Generalization
Wei Yao, Wang Zhaoyang, Gengze Xu +5
The paradigm of Weak-to-Strong Generalization (W2SG) suggests that a pre-trained strong model can surpass its weak supervisor, yet the decisive role of pre-training remains theoret…
MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging
Fanshuang Kong, Richong Zhang, Zhijie Nie +4
Model merging offers a scalable alternative to multi-task learning but often yields suboptimal performance on classification tasks. We attribute this degradation to a geometric mis…
On SkipGram Word Embedding Models with Negative Sampling: Unified Framework and Impact of Noise Distributions
Dezhi Liu, Richong Zhang, Ziqiao Wang
SkipGram word embedding models with negative sampling, or SGN in short, is an elegant family of word embedding models. In this paper, we formulate a framework for word embedding, r…
Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach
Haiyun He, Yepeng Liu, Ziqiao Wang +2
Watermarking has emerged as a crucial method to distinguish AI-generated text from human-created text. Current watermarking approaches often lack formal optimality guarantees or ad…
Distributional Information Embedding: A Framework for Multi-bit Watermarking
Haiyun He, Yepeng Liu, Ziqiao Wang +2
This paper introduces a novel problem, distributional information embedding, motivated by the practical demands of multi-bit watermarking for large language models (LLMs). Unlike t…