6 papers
Surprise Masking: Private Data Filtering with Model Likelihoods
Riya Tyagi, Sahra Ghalebikesabi, Vinnie Monaco +2
Data useful for language model (LM) training can contain private or unwanted information that is at risk of being learned or memorized. We propose Surprise Masking (SM), a simple f…
Unmasking the Secret Code: Using Canaries to Show How Masked Context Controls Memorization in SFT
Avi Schwarzschild, Om Thakkar, Sahra Ghalebikesabi +2
Canaries, or synthetic training records containing known secrets, are widely used to assess whether language models reveal information from their training data. But how much do the…
Differentially Private Parameter-Efficient Fine-tuning for Large ASR Models
Hongbin Liu, Lun Wang, Om Thakkar +2
Large ASR models can inadvertently leak sensitive information, which can be mitigated by formal privacy measures like differential privacy (DP). However, traditional DP training is…
Training Large ASR Encoders with Differential Privacy
Geeticka Chauhan, Steve Chien, Om Thakkar +2
Self-supervised learning (SSL) methods for large speech models have proven to be highly effective at ASR. With the interest in public deployment of large pre-trained models, there…
Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clipping
Lun Wang, Om Thakkar, Zhong Meng +3
Gradient clipping plays a vital role in training large-scale automatic speech recognition (ASR) models. It is typically applied to minibatch gradients to prevent gradient explosion…
Noise Masking Attacks and Defenses for Pretrained Speech Models
Matthew Jagielski, Om Thakkar, Lun Wang
Speech models are often trained on sensitive data in order to improve model performance, leading to potential privacy leakage. Our work considers noise masking attacks, introduced…