178 citations · 347 across the 23 of their papers we have counts for
26 papers
Defending Against Prompt Injection With a Few DefensiveTokens
Sizhe Chen, Yizhu Wang, Nicholas Carlini +2
When large language model (LLM) systems interact with external data to perform complex tasks, a new attack, namely prompt injection, becomes a significant threat. By injecting inst…
Does More Inference-Time Compute Really Help Robustness?
Tong Wu, Chong Xiang, Jiachen T. Wang +4
Recently, Zaremba et al. demonstrated that increasing inference-time computation improves robustness in large proprietary reasoning LLMs. In this paper, we first show that smaller-…
How much do language models memorize?
John X. Morris, Chawin Sitawarin, Chuan Guo +5
We propose a new method for estimating how much a model knows about a datapoint and use it to measure the capacity of modern language models. Prior studies of language model memori…
Lessons from Defending Gemini Against Indirect Prompt Injections
Chongyang Shi, Sharon Lin, Shuang Song +11
Gemini is increasingly used to perform tasks on behalf of users, where function-calling and tool-use capabilities enable the model to access user data. Some tools, however, require…
Vulnerability Detection with Code Language Models: How Far Are We?
Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim +6
In the context of the rising interest in code language models (code LMs) and vulnerability detection, we study the effectiveness of code LMs for detecting vulnerabilities. Our anal…
PAL: Proxy-Guided Black-Box Attack on Large Language Models
Chawin Sitawarin, Norman Mu, David Wagner +1
Large Language Models (LLMs) have surged in popularity in recent months, but they have demonstrated concerning capabilities to generate harmful content when manipulated. While tech…
StruQ: Defending Against Prompt Injection with Structured Queries
Sizhe Chen, Julien Piet, Chawin Sitawarin +1
Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capab…
Jatmo: Prompt Injection Defense by Task-Specific Finetuning
Julien Piet, Maha Alrashed, Chawin Sitawarin +5
Large Language Models (LLMs) are attracting significant research attention due to their instruction-following abilities, allowing users and developers to leverage LLMs for a variet…
Mark My Words: Analyzing and Evaluating Language Model Watermarks
Julien Piet, Chawin Sitawarin, Vivian Fang +2
The capabilities of large language models have grown significantly in recent years and so too have concerns about their misuse. It is important to be able to distinguish machine-ge…
PubDef: Defending Against Transfer Attacks From Public Models
Chawin Sitawarin, Jaewon Chang, David Huang +2
Adversarial attacks have been a looming and unaddressed threat in the industry. However, through a decade-long history of the robustness evaluation literature, we have learned that…
Positional Embedding-Aware Activations
Kathan Shah, Chawin Sitawarin
We present a neural network architecture designed to naturally learn a positional embedding and overcome the spectral bias towards lower frequencies faced by conventional activatio…
REAP: A Large-Scale Realistic Adversarial Patch Benchmark
Nabeel Hingun, Chawin Sitawarin, Jerry Li +1
Machine learning models are known to be susceptible to adversarial perturbation. One famous attack is the adversarial patch, a sticker with a particularly crafted pattern that make…
Preprocessors Matter! Realistic Decision-Based Attacks on Machine Learning Systems
Chawin Sitawarin, Florian Tramèr, Nicholas Carlini
Decision-based attacks construct adversarial examples against a machine learning (ML) model by making only hard-label queries. These attacks have mainly been applied directly to st…
Part-Based Models Improve Adversarial Robustness
Chawin Sitawarin, Kornrapat Pongmala, Yizheng Chen +2
We show that combining human prior knowledge with end-to-end learning can improve the robustness of deep neural networks by introducing a part-based model for object classification…
Demystifying the Adversarial Robustness of Random Transformation Defenses
Chawin Sitawarin, Zachary Golan-Strieb, David Wagner
Neural networks' lack of robustness against attacks raises concerns in security-sensitive settings such as autonomous vehicles. While many countermeasures may look promising, only…
Adversarial Examples for -Nearest Neighbor Classifiers Based on Higher-Order Voronoi Diagrams
Chawin Sitawarin, Evgenios M. Kornaropoulos, Dawn Song +1
Adversarial examples are a widely studied phenomenon in machine learning models. While most of the attention has been focused on neural networks, other practical models also suffer…
Minimum-Norm Adversarial Examples on KNN and KNN-Based Models
Chawin Sitawarin, David Wagner
We study the robustness against adversarial examples of kNN classifiers and classifiers that combine kNN with neural networks. The main difficulty lies in the fact that finding an…
SAT: Improving Adversarial Training via Curriculum-Based Loss Smoothing
Chawin Sitawarin, Supriyo Chakraborty, David Wagner
Adversarial training (AT) has become a popular choice for training robust networks. However, it tends to sacrifice clean accuracy heavily in favor of robustness and suffers from a…
Defending Against Adversarial Examples with K-Nearest Neighbor
Chawin Sitawarin, David Wagner
Robustness is an increasingly important property of machine learning models as they become more and more prevalent. We propose a defense against adversarial examples based on a k-n…
Better the Devil you Know: An Analysis of Evasion Attacks using Out-of-Distribution Adversarial Examples
Vikash Sehwag, Arjun Nitin Bhagoji, Liwei Song +4
A large body of recent work has investigated the phenomenon of evasion attacks using adversarial examples for deep learning systems, where the addition of norm-bounded perturbation…
On the Robustness of Deep K-Nearest Neighbors
Chawin Sitawarin, David Wagner
Despite a large amount of attention on adversarial examples, very few works have demonstrated an effective defense against this threat. We examine Deep k-Nearest Neighbor (DkNN), a…
DARTS: Deceiving Autonomous Cars with Toxic Signs
Chawin Sitawarin, Arjun Nitin Bhagoji, Arsalan Mosenia +2
Sign recognition is an integral part of autonomous cars. Any misclassification of traffic signs can potentially lead to a multitude of disastrous consequences, ranging from a life-…
Rogue Signs: Deceiving Traffic Sign Recognition with Malicious Ads and Logos
Chawin Sitawarin, Arjun Nitin Bhagoji, Arsalan Mosenia +2
We propose a new real-world attack against the computer vision based systems of autonomous vehicles (AVs). Our novel Sign Embedding attack exploits the concept of adversarial examp…
Beyond Grand Theft Auto V for Training, Testing and Enhancing Deep Learning in Self Driving Cars
Mark Martinez, Chawin Sitawarin, Kevin Finch +3
As an initial assessment, over 480,000 labeled virtual images of normal highway driving were readily generated in Grand Theft Auto V's virtual environment. Using these images, a CN…