9 papers
Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran +1
Post-training LLMs with RLHF and preference optimization methods (e.g., DPO, IPO) has greatly improved alignment, yet these approaches assume a single objective. In reality, humans…
Formalize, Don't Optimize: The Heuristic Trap in LLM-Generated Combinatorial Solvers
Haoyu Wang, Yuliang Song, Tao Li +5
Large Language Models (LLMs) struggle to solve complex combinatorial problems through direct reasoning, so recent neuro-symbolic systems increasingly use them to synthesize executa…
Best Policy Learning from Trajectory Preference Feedback
Akhil Agnihotri, Rahul Jain, Deepak Ramachandran +1
Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful approach for aligning generative models, but its reliance on learned reward models makes it vulnerable t…
Robust LLM Alignment via Distributionally Robust Direct Preference Optimization
Zaiyan Xu, Sushil Vemuri, Kishan Panaganti +3
A major challenge in aligning large language models (LLMs) with human preferences is the issue of distribution shift. LLM alignment algorithms rely on static preference datasets, a…
Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Concepts
Ibtihel Amara, Ahmed Imtiaz Humayun, Ivana Kajic +12
Concept erasure techniques have recently gained significant attention for their potential to remove unwanted concepts from text-to-image models. While these methods often demonstra…
Preference Adaptive and Sequential Text-to-Image Generation
Ofir Nabati, Guy Tennenholtz, ChihWei Hsu +5
We address the problem of interactive text-to-image (T2I) generation, designing a reinforcement learning (RL) agent which iteratively improves a set of generated images for a user…