8 papers
How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?
Prakhar Gupta, Terry Jingchen Zhang, Florent Draye +2
Modern LLMs are alarmingly susceptible to surprisingly simple immaterial changes of input prompts: a casual hint, an incorrectly labeled few-shot example, or a fake prior assistant…
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
Prakhar Gupta, Garv Shah, Satyam Goyal +1
Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) t…
Self-Mined Hardness for Safety Fine-Tuning
Prakhar Gupta, Garv Shah, Donghua Zhang
Safety fine-tuning of language models typically requires a curated adversarial dataset. We take a different approach: score each candidate prompt's difficulty by how often the targ…
Consistency Training Along the Transformer Stack
Sukrati Gautam, Neil Shah, Arav Dhoot +7
Consistency training encourages models to behave similarly across different contexts, and has shown promise for reducing misalignment. We broaden the scope of consistency training…
Consistency Training while Mitigating Obfuscation via Rate Matching
Sohaib Imran, Prakhar Gupta, Jannes Elstner +1
Large language models are often influenced by extraneous input features, such as cues revealing a user's preferred answer. Consistency training reduces this influence by training m…
Improving Sparse Memory Finetuning
Satyam Goyal, Anirudh Kanchi, Garv Shah +1
Large Language Models (LLMs) are typically static after training, yet real-world applications require continual adaptation to new knowledge without degrading existing capabilities.…