7 papers · 1 filter
Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran +1
Post-training LLMs with RLHF and preference optimization methods (e.g., DPO, IPO) has greatly improved alignment, yet these approaches assume a single objective. In reality, humans…
Best Policy Learning from Trajectory Preference Feedback
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran +1
Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful approach for aligning generative models, but its reliance on learned reward models makes it vulnerable t…
Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
Zaiyan Xu, Sushil Vemuri, Kishan Panaganti +3
A major challenge in aligning large language models (LLMs) with human preferences is the issue of distribution shift. LLM alignment algorithms rely on static preference datasets, a…
Online Bandit Learning with Offline Preference Data for Improved RLHF
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran +1
Reinforcement Learning with Human Feedback (RLHF) is at the core of fine-tuning methods for generative AI models for language and images. Such feedback is often sought as rank or p…
e-COP : Episodic Constrained Optimization of Policies
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran +1
In this paper, we present the algorithm, the first policy optimization algorithm for constrained Reinforcement Learning (RL) in episodic (finite horizon) settings.…
Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation
Katherine M. Collins, Najoung Kim, Yonatan Bitton +15
Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for learning an accurate rew…