1.6k citations · 2.2k across the 52 of their papers we have counts for
13 papers · 1 filter
The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections
Milad Nasr, Nicholas Carlini, Chawin Sitawarin +11
How should we evaluate the robustness of language model defenses? Current defenses against jailbreaks and prompt injections (which aim to prevent an attacker from eliciting harmful…
Cascading Adversarial Bias from Injection to Distillation in Language Models
Harsh Chaudhari, Jamie Hayes, Matthew Jagielski +3
Model distillation has become essential for creating smaller, deployable language models that retain larger system capabilities. However, widespread deployment raises concerns abou…
Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards
Yangsibo Huang, Milad Nasr, Anastasios Angelopoulos +10
It is now common to evaluate Large Language Models (LLMs) by having humans manually vote to evaluate model outputs, in contrast to typical benchmarks that evaluate knowledge or ski…
Avoiding Generative Model Writer's Block With Embedding Nudging
Ali Zand, Milad Nasr
Generative image models, since introduction, have become a global phenomenon. From new arts becoming possible to new vectors of abuse, many new capabilities have become available.…
Private Fine-tuning of Large Language Models with Zeroth-order Optimization
Xinyu Tang, Ashwinee Panda, Milad Nasr +2
Differentially private stochastic gradient descent (DP-SGD) allows models to be trained in a privacy-preserving manner, but has proven difficult to scale to the era of foundation m…
Scalable Extraction of Training Data from (Production) Language Models
Milad Nasr, Nicholas Carlini, Jonathan Hayase +7
This paper studies extractable memorization: training data that an adversary can efficiently extract by querying a machine learning model without prior knowledge of the training da…
Reverse-Engineering Decoding Strategies Given Blackbox Access to a Language Generation System
Daphne Ippolito, Nicholas Carlini, Katherine Lee +2
Neural language models are increasingly deployed into APIs and websites that allow a user to pass in a prompt and receive generated text. Many of these systems do not reveal genera…
Privacy Auditing with One (1) Training Run
Thomas Steinke, Milad Nasr, Matthew Jagielski
We propose a scheme for auditing differentially private machine learning systems with a single training run. This exploits the parallelism of being able to add or remove multiple t…
Why Is Public Pretraining Necessary for Private Model Training?
Arun Ganesh, Mahdi Haghifam, Milad Nasr +5
In the privacy-utility tradeoff of a model trained on benchmark language and vision tasks, remarkable improvements have been widely reported with the use of pretraining on publicly…
Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy
Daphne Ippolito, Florian Tramèr, Milad Nasr +5
Studying data memorization in neural language models helps us understand the risks (e.g., to privacy or copyright) associated with models regurgitating training data and aids in th…
No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"
Nicholas Carlini, Vitaly Feldman, Milad Nasr
New methods designed to preserve data privacy require careful scrutiny. Failure to preserve privacy is hard to detect, and yet can lead to catastrophic results when a system implem…
Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning
Milad Nasr, Shuang Song, Abhradeep Thakurta +2
Differentially private (DP) machine learning allows us to train models on private data while limiting data leakage. DP formalizes this data leakage through a cryptographic game, wh…
Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising
Milad Nasr, Reza Shokri, Amir houmansadr
Deep learning models leak significant amounts of information about their training datasets. Previous work has investigated training models with differential privacy (DP) guarantees…