Surprise Masking: Private Data Filtering with Model Likelihoods

Riya Tyagi, Sahra Ghalebikesabi, Vinnie Monaco +2

Data useful for language model (LM) training can contain private or unwanted information that is at risk of being learned or memorized. We propose Surprise Masking (SM), a simple f…