164 citations · 527 across the 37 of their papers we have counts for
42 papers
Surprise Masking: Private Data Filtering with Model Likelihoods
Riya Tyagi, Sahra Ghalebikesabi, Vinnie Monaco +2
Data useful for language model (LM) training can contain private or unwanted information that is at risk of being learned or memorized. We propose Surprise Masking (SM), a simple f…
Exploring The Empirical Privacy Landscape of Synthetic Data
Zihao Zhao, Avi Schwarzschild, Casey Meehan +3
Methods for training language models on sensitive data differ in how they balance capability gains against the risk of leaking private information. In this paper, we present a cont…
Unmasking the Secret Code: Using Canaries to Show How Masked Context Controls Memorization in SFT
Avi Schwarzschild, Om Thakkar, Sahra Ghalebikesabi +2
Canaries, or synthetic training records containing known secrets, are widely used to assess whether language models reveal information from their training data. But how much do the…
Model Card for OpenAI Privacy Filter
Charles de Bourcy, Sahra Ghalebikesabi, Avi Schwarzschild +22
OpenAI Privacy Filter is a compact, bidirectional token-classification model for detecting and redacting personally identifiable information (PII) and secrets in unstructured text.…
Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
Sean McLeish, Ang Li, John Kirchenbauer +7
Recent advances in depth-recurrent language models show that recurrence can decouple train-time compute and parameter count from test-time compute. In this work, we study how to co…
Toward Understanding the Transferability of Adversarial Suffixes in Large Language Models
Sarah Ball, Niki Hasrati, Alexander Robey +4
Discrete optimization-based jailbreaking attacks on large language models aim to generate short, nonsensical suffixes that, when appended onto input prompts, elicit disallowed cont…
Command-V: Pasting LLM Behaviors via Activation Profiles
Barry Wang, Avi Schwarzschild, Alexander Robey +4
Retrofitting large language models (LLMs) with new behaviors typically requires full finetuning or distillation-costly steps that must be repeated for every architecture. In this w…
Extrapolation by Association: Length Generalization Transfer in Transformers
Ziyang Cai, Nayoung Lee, Avi Schwarzschild +2
Transformer language models have demonstrated impressive generalization capabilities in natural language domains, yet we lack a fine-grained understanding of how such generalizatio…
Existing Large Language Model Unlearning Evaluations Are Inconclusive
Zhili Feng, Yixuan Even Xu, Alexander Robey +5
Machine unlearning aims to remove sensitive or undesired data from large language models. However, recent studies suggest that unlearning is often shallow, claiming that removed kn…
RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models
Bang An, Shiyue Zhang, Mark Dredze
Efforts to ensure the safety of large language models (LLMs) include safety fine-tuning, evaluation, and red teaming. However, despite the widespread use of the Retrieval-Augmented…
Antidistillation Sampling
Yash Savani, Asher Trockman, Zhili Feng +5
Frontier models that generate extended reasoning traces inadvertently produce rich token sequences that can facilitate model distillation. Recognizing this vulnerability, model own…
Has My System Prompt Been Used? Large Language Model Prompt Membership Inference
Roman Levin, Valeriia Cherepanova, Abhimanyu Hans +2
Prompt engineering has emerged as a powerful technique for optimizing large language models (LLMs) for specific applications, enabling faster prototyping and improved performance,…
Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges
Nayoung Lee, Ziyang Cai, Avi Schwarzschild +2
Large language models often struggle with length generalization and solving complex problem instances beyond their training distribution. We present a self-improvement approach whe…
Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization
Mucong Ding, Chenghao Deng, Jocelyn Choo +8
While generalization over tasks from easy to hard is crucial to profile language models (LLMs), the datasets with fine-grained difficulty annotations for each problem across a broa…
Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers
Joshua Nathaniel Williams, Avi Schwarzschild, Yutong He +1
Recovering natural language prompts for image generation models, solely based on the generated images is a difficult discrete optimization problem. In this work, we present the fir…
The CLRS-Text Algorithmic Reasoning Language Benchmark
Larisa Markeeva, Sean McLeish, Borja Ibarz +7
Eliciting reasoning capabilities from language models (LMs) is a critical direction on the path towards building intelligent systems. Most recent studies dedicated to reasoning foc…
Transformers Can Do Arithmetic with the Right Embeddings
Sean McLeish, Arpit Bansal, Alex Stein +8
The poor performance of transformers on arithmetic tasks seems to stem in large part from their inability to keep track of the exact position of each digit inside of a large span o…
Benchmarking ChatGPT on Algorithmic Reasoning
Sean McLeish, Avi Schwarzschild, Tom Goldstein
We evaluate ChatGPT's ability to solve algorithm problems from the CLRS benchmark suite that is designed for GNNs. The benchmark requires the use of a specified classical algorithm…
Rethinking LLM Memorization through the Lens of Adversarial Compression
Avi Schwarzschild, Zhili Feng, Pratyush Maini +2
Large language models (LLMs) trained on web-scale datasets raise substantial concerns regarding permissible data usage. One major question is whether these models "memorize" all th…
Forcing Diffuse Distributions out of Language Models
Yiming Zhang, Avi Schwarzschild, Nicholas Carlini +2
Despite being trained specifically to follow user instructions, today's instructiontuned language models perform poorly when instructed to produce random outputs. For example, when…
TOFU: A Task of Fictitious Unlearning for LLMs
Pratyush Maini, Zhili Feng, Avi Schwarzschild +2
Large language models trained on massive corpora of data from the web can memorize and reproduce sensitive or private data raising both legal and ethical concerns. Unlearning, or t…
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
Abhimanyu Hans, Avi Schwarzschild, Valeriia Cherepanova +5
Detecting text generated by modern large language models is thought to be hard, as both LLMs and humans can exhibit a wide range of complex behaviors. However, we find that a score…
Effective Backdoor Mitigation in Vision-Language Models Depends on the Pre-training Objective
Sahil Verma, Gantavya Bhatt, Avi Schwarzschild +6
Despite the advanced capabilities of contemporary machine learning (ML) models, they remain vulnerable to adversarial and backdoor attacks. This vulnerability is particularly conce…
NEFTune: Noisy Embeddings Improve Instruction Finetuning
Neel Jain, Ping-yeh Chiang, Yuxin Wen +10
We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard fi…