11 citations · 24 across the 10 of their papers we have counts for
10 papers
Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
Seungju Han, Konwoo Kim, Chanwoo Park +5
Synthetic data augmentation helps language models learn new knowledge in data-constrained domains. However, naively scaling existing synthetic data methods by training on more synt…
Data-efficient pre-training by scaling synthetic megadocs
Konwoo Kim, Suhas Kotha, Yejin Choi +3
Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that…
Replaying pre-training data improves fine-tuning
Suhas Kotha, Percy Liang
To obtain a language model for a target domain (e.g. math), the current paradigm is to pre-train on a vast amount of generic web text and then fine-tune on the relatively limited a…
Pre-training under infinite compute
Konwoo Kim, Suhas Kotha, Percy Liang +1
Since compute grows much faster than web text available for language model pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We…
A Safe Harbor for AI Evaluation and Red Teaming
Shayne Longpre, Sayash Kapoor, Kevin Klyman +20
Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems. However, the terms of service and enforcement strategies used by promi…
Testing the Limits of Jailbreaking Defenses with the Purple Problem
Taeyoun Kim, Suhas Kotha, Aditi Raghunathan
The rise of "jailbreak" attacks on language models has led to a flurry of defenses aimed at preventing undesirable responses. We critically examine the two stages of the defense pi…
Repetition Improves Language Model Embeddings
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried +2
Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text embedding models have lar…
Understanding Catastrophic Forgetting in Language Models via Implicit Inference
Suhas Kotha, Jacob Mitchell Springer, Aditi Raghunathan
We lack a systematic understanding of the effects of fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback), particularly on tasks outsi…
Provably Bounding Neural Network Preimages
Suhas Kotha, Christopher Brix, Zico Kolter +2
Most work on the formal verification of neural networks has focused on bounding the set of outputs that correspond to a given set of inputs (for example, bounded perturbations of a…
CELESTIAL: Classification Enabled via Labelless Embeddings with Self-supervised Telescope Image Analysis Learning
Suhas Kotha, Anirudh Koul, Siddha Ganju +1
A common class of problems in remote sensing is scene classification, a fundamentally important task for natural hazards identification, geographic image retrieval, and environment…