Can You Keep a Secret? Involuntary Information Leakage in Language Model Writing

Ari Holtzman, Peter West

Language models are deployed in settings that require compartmentalization: system prompts should not be disclosed, chain-of-thought reasoning is hidden from users, and sensitive d…