11 citations · 24 across the 10 of their papers we have counts for
3 papers · 1 filter
Replaying pre-training data improves fine-tuning
Suhas Kotha, Percy Liang
To obtain a language model for a target domain (e.g. math), the current paradigm is to pre-train on a vast amount of generic web text and then fine-tune on the relatively limited a…
Repetition Improves Language Model Embeddings
Jacob Mitchell Springer, Suhas Kotha, Daniel Fried +2
Bidirectional models are considered essential for strong text embeddings. Recent approaches to adapt autoregressive language models (LMs) into strong text embedding models have lar…
Understanding Catastrophic Forgetting in Language Models via Implicit Inference
Suhas Kotha, Jacob Mitchell Springer, Aditi Raghunathan
We lack a systematic understanding of the effects of fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback), particularly on tasks outsi…