273 citations · 1.4k across the 55 of their papers we have counts for
27 papers · 1 filter
Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks
Aashiq Muhamed, Mona T. Diab, Virginia Smith +2
Backdoor poisoning attacks add poisoned examples to otherwise-clean finetuning data, pairing a trigger with a target behavior that the model learns to produce when the trigger appe…
TabPATE: Differentially Private Tabular In-Context Learning Without Public Data
Dariush Wahdany, Matthew Jagielski, Jesse C. Cresswell +2
Tabular foundation models enable accurate in-context learning (ICL) from small labeled datasets, but the private records placed in context can leak through model predictions. We fi…
Curation Leaks: Membership Inference Attacks against Data Curation for Machine Learning
Dariush Wahdany, Matthew Jagielski, Adam Dziedzic +1
In machine learning, curation is used to select the most valuable data for improving both model accuracy and computational efficiency. Recently, curation has also been explored as…
SoK: Data Minimization in Machine Learning
Robin Staab, Nikola Jovanović, Kimberly Mai +4
Data minimization (DM) describes the principle of collecting only the data strictly necessary for a given task. It is a foundational principle across major data protection regulati…
Black-Box Privacy Attacks on Shared Representations in Multitask Learning
John Abascal, Nicolás Berrios, Alina Oprea +3
Multitask learning (MTL) has emerged as a powerful paradigm that leverages similarities among multiple learning tasks, each with insufficient samples to train a standalone model, t…
Cascading Adversarial Bias from Injection to Distillation in Language Models
Harsh Chaudhari, Jamie Hayes, Matthew Jagielski +3
Model distillation has become essential for creating smaller, deployable language models that retain larger system capabilities. However, widespread deployment raises concerns abou…
Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards
Yangsibo Huang, Milad Nasr, Anastasios Angelopoulos +10
It is now common to evaluate Large Language Models (LLMs) by having humans manually vote to evaluate model outputs, in contrast to typical benchmarks that evaluate knowledge or ski…
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research
A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen +34
"Machine unlearning" is a popular proposed solution for mitigating the existence of content in an AI model that is problematic for legal or moral reasons, including privacy, copyri…
UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI
Ilia Shumailov, Jamie Hayes, Eleni Triantafillou +6
Exact unlearning was first introduced as a privacy mechanism that allowed a user to retract their data from machine learning models on request. Shortly after, inexact schemes were…
Differentially Private Prototypes for Imbalanced Transfer Learning
Dariush Wahdany, Matthew Jagielski, Adam Dziedzic +1
Machine learning (ML) models have been shown to leak private information from their training datasets. Differential Privacy (DP), typically implemented through the differential pri…
Noise Masking Attacks and Defenses for Pretrained Speech Models
Matthew Jagielski, Om Thakkar, Lun Wang
Speech models are often trained on sensitive data in order to improve model performance, leading to potential privacy leakage. Our work considers noise masking attacks, introduced…
Scalable Extraction of Training Data from (Production) Language Models
Milad Nasr, Nicholas Carlini, Jonathan Hayase +7
This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training da…
Privacy Auditing with One (1) Training Run
Thomas Steinke, Milad Nasr, Matthew Jagielski
We propose a scheme for auditing differentially private machine learning systems with a single training run. This exploits the parallelism of being able to add or remove multiple t…
Randomness in ML Defenses Helps Persistent Attackers and Hinders Evaluators
Keane Lucas, Matthew Jagielski, Florian Tramèr +2
It is becoming increasingly imperative to design robust ML defenses. However, recent work has found that many defenses that initially resist state-of-the-art attacks can be broken…
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy
Daphne Ippolito, Florian Tramèr, Milad Nasr +5
Studying data memorization in neural language models helps us understand the risks (e.g., to privacy or copyright) associated with models regurgitating training data and aids in th…
SNAP: Efficient Extraction of Private Properties with Poisoning
Harsh Chaudhari, John Abascal, Alina Oprea +3
Property inference attacks allow an adversary to extract global properties of the training dataset from a machine learning model. Such attacks have privacy implications for data ow…
Measuring Forgetting of Memorized Training Examples
Matthew Jagielski, Om Thakkar, Florian Tramèr +8
Machine learning models exhibit two seemingly contradictory phenomena: training data memorization, and various forms of forgetting. In memorization, models overfit specific trainin…
The Privacy Onion Effect: Memorization is Relative
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang +3
Machine learning models trained on private datasets have been shown to leak their private data. While recent work has found that the average data point is rarely leaked, the outlie…
How to Combine Membership-Inference Attacks on Multiple Updated Models
Matthew Jagielski, Stanley Wu, Alina Oprea +2
A large body of research has shown that machine learning models are vulnerable to membership inference (MI) attacks that violate the privacy of the participants in the training dat…
Subverting Fair Image Search with Generative Adversarial Perturbations
Avijit Ghosh, Matthew Jagielski, Christo Wilson
In this work we explore the intersection fairness and robustness in the context of ranking: when a ranking model has been calibrated to achieve some definition of fairness, is it p…
Debugging Differential Privacy: A Case Study for Privacy Auditing
Florian Tramer, Andreas Terzis, Thomas Steinke +3
Differential Privacy can provide provable privacy guarantees for training data in machine learning. However, the presence of proofs does not preclude the presence of errors. Inspir…
Quantifying Memorization Across Neural Language Models
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski +3
Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is…
Subpopulation Data Poisoning Attacks
Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger +1
Machine learning systems are deployed in critical settings, but they might fail in unexpected ways, impacting the accuracy of their predictions. Poisoning attacks against machine l…
Cryptanalytic Extraction of Neural Network Models
Nicholas Carlini, Matthew Jagielski, Ilya Mironov
We argue that the machine learning problem of model extraction is actually a cryptanalytic problem in disguise, and should be studied as such. Given oracle access to a neural netwo…