machine learning

Watermark Forensics for Generative Models: An Information-Theoretic Perspective

arXiv:2607.13003

summary

The paper studies how to embed and detect watermarks in the outputs of generative language models, providing an information‑theoretic analysis of the trade‑offs for detection, attribution, payload extraction, and localization, and derives tight entropy‑rate bounds validated on GPT‑2, Pythia, and Qwen2.5.

Abstract

A watermark in a generative model's output is usually asked only whether a text is machine-made. The same mark can do more: attribute it to the user who produced it, extract a hidden payload, or localize the part that survives editing. These form a forensic ladder, and we ask what each rung costs in the sample length $n$ . One object organizes the answers. Let $S$ be the secret the mark carries (a user's identity or payload), and let the information profile $I^½ (t) = I (S; X_{t} ∣ X_{< t})$ record how much the $t$ -th token reveals about $S$ given the earlier ones. Its total mass pays for attribution and extraction; how that mass is spread pays for localization; and detection alone is paid for not by information but by presence, the distance from the marked to the unmarked distribution. The literature's two quality models, a mark subtle on every token and one that stamps a few tokens loudly, are two incomparable ways of capping this profile. Our main theorem settles the ladder's entropy column. For statistically distortion-free schemes, attributing a text to one of $N$ users costs $I^ (lo g N / h)$ tokens over every stationary-ergodic source of entropy rate $h$ , sharp to a $(1 + o (1))$ factor: to our knowledge the first tight entropy-rate law for multi-user attribution (via exact alignment). The natural collision-counting analysis overcharges without bound; only a decoder thresholding each candidate by its own realized surprisal attains the rate while almost never implicating an innocent user. A matching converse makes the law two-sided, and extraction of an $ℓ$ -bit payload costs $I^ (ℓ / h)$ . Two gaps are real, not modeling artifacts: a $I^ (lo g N)$ -token window in which a text is provably machine-made yet unattributable, and a footprint-resolution uncertainty principle. Experiments on GPT-2, Pythia-410M, and Qwen2.5 recover the predicted constants.

The abstract has been shortened to comply with arXiv's length limit

Topics & keywords

#watermarking #generative models #information theory #attribution #payload extraction #localizationwatermarkgenerative language modelinformation profileentropy-rate lawsurprisal decodingcollision countingGPT-2Pythia-410MQwen2.5