74 citations · 81 across the 10 of their papers we have counts for
11 papers
Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model
Jacqueline He, Jonathan Hayase, Wen-tau Yih +3
Language models (LMs) tend to memorize portions of their training data and emit verbatim spans. When the underlying sources are sensitive or copyright-protected, such reproduction…
Sampling from Your Language Model One Byte at a Time
Jonathan Hayase, Alisa Liu, Noah A. Smith +1
Tokenization is used almost universally by modern language models, enabling efficient text representation using multi-byte or multi-character tokens. However, prior work has shown…
SuperBPE: Space Travel for Language Models
Alisa Liu, Jonathan Hayase, Valentin Hofmann +3
The assumption across nearly all language model (LM) tokenization schemes is that tokens should be subwords, i.e., contained within word boundaries. While providing a seemingly rea…
Scalable Fingerprinting of Large Language Models
Anshul Nasery, Jonathan Hayase, Creston Brooks +4
Model fingerprinting has emerged as a powerful tool for model owners to identify their shared model given API access. However, to lower false discovery rate, fight fingerprint leak…
OML: A Primitive for Reconciling Open Access with Owner Control in AI Model Distribution
Zerui Cheng, Edoardo Contente, Ben Finch +9
The current paradigm of AI model distribution presents a fundamental dichotomy: models are either closed and API-gated, sacrificing transparency and local execution, or openly dist…
Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
Jonathan Hayase, Alisa Liu, Yejin Choi +2
The pretraining data of today's strongest language models is opaque; in particular, little is known about the proportions of various domains or languages represented. In this work,…
Insufficient Statistics Perturbation: Stable Estimators for Private Least Squares
Gavin Brown, Jonathan Hayase, Samuel Hopkins +5
We present a sample- and time-efficient differentially private algorithm for ordinary least squares, with error that depends linearly on the dimension and is independent of the con…
Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion
Shuqi Ke, Charlie Hou, Sewoong Oh +1
We show that differentially private full fine-tuning (DP-FFT) can distort pre-trained backbone features based on both theoretical and empirical results. We identify the cause of th…
Label Poisoning is All You Need
Rishi D. Jha, Jonathan Hayase, Sewoong Oh
In a backdoor attack, an adversary injects corrupted data into a model's training dataset in order to gain control over its predictions on images with a specific attacker-defined t…
DataComp: In search of the next generation of multimodal datasets
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang +31
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model arch…
Towards a Defense Against Federated Backdoor Attacks Under Continuous Training
Shuaiqi Wang, Jonathan Hayase, Giulia Fanti +1
Backdoor attacks are dangerous and difficult to prevent in federated learning (FL), where training data is sourced from untrusted clients over long periods of time. These difficult…