2 citations · 5 across the 16 of their papers we have counts for
18 papers
Speculative Probing: LLM Monitoring at Speculative-Decoding Cost
Collin Zhang, Tingwei Zhang, Vitaly Shmatikov
Real-time classification during language model inference is valuable for safety filtering, behavioral analysis, and model monitoring, but current approaches force a trade-off betwe…
Agent Meltdowns: The Road to Hell Is Paved with Helpful Agents
Rishi Jha, Harold Triedman, Arkaprabha Bhattacharya +1
Agents operating with computer and Web use inevitably encounter errors: inaccessible webpages, missing files, local and remote misconfigurations, etc. These errors do not thwart ag…
Deep-Research Agents Can Be Poisoned via User-Generated Content
Tingwei Zhang, Harold Triedman, Vitaly Shmatikov
Deep-research agents are an alternative to conventional Web search. They use multi-agent pipelines to issue multiple Web searches related to user queries, retrieve relevant content…
How to Steal Reasoning Without Reasoning Traces
Tingwei Zhang, John X. Morris, Vitaly Shmatikov
Many large language models (LLMs) use reasoning to generate responses but do not reveal their full reasoning traces (a.k.a. chains of thought), instead outputting only final answer…
Learning to Detect Language Model Training Data via Active Reconstruction
Junjie Oscar Yin, John X. Morris, Vitaly Shmatikov +2
Detecting LLM training data is generally framed as a membership inference attack (MIA) problem. However, conventional MIAs operate passively on fixed model weights, using log-likel…
Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems
Rishi Jha, Harold Triedman, Justin Wagle +1
Control-flow hijacking attacks manipulate orchestration mechanisms in multi-agent systems into performing unsafe actions that compromise the system and exfiltrate sensitive informa…
MillStone: How Open-Minded Are LLMs?
Harold Triedman, Vitaly Shmatikov
Large language models equipped with Web search, information retrieval tools, and other agentic capabilities are beginning to supplant traditional search engines. As users start to…
Approximating Language Model Training Data from Weights
John X. Morris, Junjie Oscar Yin, Woojeong Kim +2
Modern language models often have open weights but closed training data. We formalize the problem of data approximation from model weights and propose several baselines and metrics…
Harnessing the Universal Geometry of Embeddings
Rishi Jha, Collin Zhang, Vitaly Shmatikov +1
We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised ap…
Universal Zero-shot Embedding Inversion
Collin Zhang, John X. Morris, Vitaly Shmatikov
Embedding inversion, i.e., reconstructing text given its embedding and black-box access to the embedding encoder, is a fundamental problem in both NLP and security. From the NLP pe…
Multi-Agent Systems Execute Arbitrary Malicious Code
Harold Triedman, Rishi Jha, Vitaly Shmatikov
Multi-agent systems coordinate LLM-based agents to perform tasks on users' behalf. In real-world applications, multi-agent systems will inevitably interact with untrusted inputs, s…
Rerouting LLM Routers
Avital Shafran, Roei Schuster, Thomas Ristenpart +1
LLM routers aim to balance quality and cost of generation by classifying queries and routing them to a cheaper or more expensive LLM depending on their complexity. Routers represen…
Adversarial Hubness in Multi-Modal Retrieval
Tingwei Zhang, Fnu Suya, Rishi Jha +2
Hubness is a phenomenon in high-dimensional vector spaces where a point from the natural distribution is unusually close to many other points. This is a well-known problem in infor…
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Research
A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen +34
"Machine unlearning" is a popular proposed solution for mitigating the existence of content in an AI model that is problematic for legal or moral reasons, including privacy, copyri…
Adversarial Decoding: Generating Readable Documents for Adversarial Objectives
Collin Zhang, Tingwei Zhang, Vitaly Shmatikov
We design, implement, and evaluate adversarial decoding, a new, generic text generation technique that produces readable documents for different adversarial objectives. Prior metho…
Self-interpreting Adversarial Images
Tingwei Zhang, Collin Zhang, John X. Morris +2
We introduce a new type of indirect, cross-modal injection attacks against visual language models that enable creation of self-interpreting images. These images contain hidden "met…
Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documents
Avital Shafran, Roei Schuster, Vitaly Shmatikov
Retrieval-augmented generation (RAG) systems respond to queries by retrieving relevant documents from a knowledge database and applying an LLM to the retrieved documents. We demons…
Extracting Prompts by Inverting LLM Outputs
Collin Zhang, John X. Morris, Vitaly Shmatikov
We consider the problem of language model inversion: given outputs of a language model, we seek to extract the prompt that generated these outputs. We develop a new black-box metho…