99 citations · 183 across the 27 of their papers we have counts for
31 papers
Aligning Language Models Towards Counterfactual Invariance
Basar Demir, Emily Shen, Ksheeraj Sai Vepuri +5
Small variations in prompt characteristics can meaningfully alter the semantics, tone, and contextual assumptions of large language model outputs. Such characteristics can involve…
MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning
Sourabh Kulkarni, Ksheeraj Sai Vepuri, Basar Demir +8
Reinforcement learning (RL) is widely used in language-model post-training to maximize rewards assigned to individual model outputs, such as scores from binary verifiers or reward…
Inverse RL Helps Align AI by Imitating Humans
Michał Wiliński, Liu Leqi, Chirag Nagpal
Language model alignment aims to make model behavior reflect desirable properties such as helpfulness, safety, and instruction following. Existing methods draw supervision from exp…
Discretizing Reward Models
Vijay Viswanathan, Shiqi Wang, Devamanyu Hazarika +4
Despite their widespread use, the role of reward models in shaping reinforcement learning is poorly understood. Reward models offer a tempting promise: they automatically estimate…
Muse Spark Safety & Preparedness Report
Cristina Menghini, Peter Ney, Hamza Kwisaba +117
Muse Spark is the latest large language model developed by Meta. In this report, we first present evaluations for catastrophic risk domains under Meta's Advanced AI Scaling Framewo…
The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes
Redacted by arXiv
This document consolidates publicly reported technical details about Metas Llama 4 model family. It summarizes (i) released variants (Scout and Maverick) and the broader herd conte…
Preference Models assume Proportional Hazards of Utilities
Chirag Nagpal
Approaches for estimating preferences from human annotated data typically involves inducing a distribution over a ranked list of choices such as the Plackett-Luce model. Indeed, mo…
Understanding challenges to the interpretation of disaggregated evaluations of algorithmic fairness
Stephen R. Pfohl, Natalie Harris, Chirag Nagpal +12
Disaggregated evaluation across subgroups is critical for assessing the fairness of machine learning models, but its uncritical use can mislead practitioners. We show that equal pe…
Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models
Roberto-Rafael Maura-Rivero, Chirag Nagpal, Roma Patel +1
Current methods that train large language models (LLMs) with reinforcement learning feedback, often resort to averaging outputs of multiple rewards functions during training. This…
Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Concepts
Ibtihel Amara, Ahmed Imtiaz Humayun, Ivana Kajic +12
Concept erasure techniques have recently gained significant attention for their potential to remove unwanted concepts from text-to-image models. While these methods often demonstra…
Speech Recognition With LLMs Adapted to Disordered Speech Using Reinforcement Learning
Chirag Nagpal, Subhashini Venugopalan, Jimmy Tobin +3
We introduce a large language model (LLM) capable of processing speech inputs and show that tuning it further with reinforcement learning on human preference (RLHF) enables it to a…
InfAlign: Inference-aware language model alignment
Ananth Balashankar, Ziteng Sun, Jonathan Berant +9
Language model alignment is a critical step in training modern generative language models. Alignment targets to improve win rate of a sample from the aligned model against the base…
Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
Amrith Setlur, Chirag Nagpal, Adam Fisch +6
A promising approach for improving reasoning in large language models is to use process reward models (PRMs). PRMs provide feedback at each step of a multi-step reasoning trace, po…
Robust Preference Optimization through Reward Model Distillation
Adam Fisch, Jacob Eisenstein, Vicky Zayats +5
Language model (LM) post-training (or alignment) involves maximizing a reward function that is derived from preference annotations. Direct Preference Optimization (DPO) is a popula…
A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models
Stephen R. Pfohl, Heather Cole-Lewis, Rory Sayres +27
Large language models (LLMs) hold promise to serve complex health information needs but also have the potential to introduce harm and exacerbate health disparities. Reliably evalua…
Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluation
Kristian Lum, Jacy Reese Anthis, Kevin Robinson +2
Standard benchmarks of bias and fairness in large language models (LLMs) measure the association between the user attributes stated or implied by a prompt and the LLM's short text…
Transforming and Combining Rewards for Aligning Large Language Models
Zihao Wang, Chirag Nagpal, Jonathan Berant +4
A common approach for aligning language models to human preferences is to first learn a reward model from preference data, and then use this reward model to update the language mod…
Theoretical guarantees on the best-of-n alignment policy
Ahmad Beirami, Alekh Agarwal, Jonathan Berant +4
A simple and effective method for the inference-time alignment and scaling test-time compute of generative models is best-of- sampling, where samples are drawn from a refere…
Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal +9
Reward models play a key role in aligning language model applications towards human preferences. However, this setup creates an incentive for the language model to exploit errors i…
Recovering Sparse and Interpretable Subgroups with Heterogeneous Treatment Effects with Censored Time-to-Event Outcomes
Chirag Nagpal, Vedant Sanil, Artur Dubrawski
Studies involving both randomized experiments as well as observational data typically involve time-to-event outcomes such as time-to-failure, death or onset of an adverse condition…
Participatory Personalization in Classification
Hailey Joren, Chirag Nagpal, Katherine Heller +1
Machine learning models are often personalized with information that is protected, sensitive, self-reported, or costly to acquire. These models use information about people but do…
auton-survival: an Open-Source Package for Regression, Counterfactual Estimation, Evaluation and Phenotyping with Censored Time-to-Event Data
Chirag Nagpal, Willa Potosnak, Artur Dubrawski
Applications of machine learning in healthcare often require working with time-to-event prediction tasks including prognostication of an adverse event, re-hospitalization or death.…
Counterfactual Phenotyping with Censored Time-to-Events
Chirag Nagpal, Mononito Goswami, Keith Dufendach +1
Estimation of treatment efficacy of real-world clinical interventions involves working with continuous outcomes such as time-to-death, re-hospitalization, or a composite event that…
Deep Cox Mixtures for Survival Regression
Chirag Nagpal, Steve Yadlowsky, Negar Rostamzadeh +1
Survival analysis is a challenging variation of regression modeling because of the presence of censoring, where the outcome measurement is only partially known, due to, for example…