217 citations · 331 across the 28 of their papers we have counts for
14 papers · 1 filter
Displacement-Resistant Extensions of DPO with Nonconvex -Divergences
Idan Pipano, Shoham Sabach, Kavosh Asadi +1
DPO and related algorithms align language models by directly optimizing the RLHF objective: find a policy that maximizes the Bradley-Terry reward while staying close to a reference…
Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains
Luca Viano, Ruida Zhou, Yifan Sun +4
The class of direct preference optimization (DPO) algorithms has emerged as a promising approach for solving the alignment problem in foundation models. These algorithms work with…
Directional-Clamp PPO
Gilad Karpel, Ruida Zhou, Shoham Sabach +1
Proximal Policy Optimization (PPO) is widely regarded as one of the most successful deep reinforcement learning algorithms, known for its robustness and effectiveness across a rang…
FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain
Rohan Deb, Kiran Thekumparampil, Kousha Kalantari +3
Supervised fine-tuning (SFT) is a standard approach to adapting large language models (LLMs) to new domains. In this work, we improve the statistical efficiency of SFT by selecting…
C2-DPO: Constrained Controlled Direct Preference Optimization
Kavosh Asadi, Julien Han, Idan Pipano +5
Direct preference optimization (\texttt{DPO}) has emerged as a promising approach for solving the alignment problem in AI. In this paper, we make two counter-intuitive observations…
ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
Hongyi Liu, Rajarshi Saha, Zhen Jia +5
Large Language Models (LLMs) have demonstrated exceptional performance in natural language processing tasks, yet their massive size makes serving them inefficient and costly. Semi-…
A Proximal Operator for Inducing 2:4-Sparsity
Jonas M Kübler, Yu-Xiang Wang, Shoham Sabach +5
Recent hardware advancements in AI Accelerators and GPUs allow to efficiently compute sparse matrix multiplications, especially when 2 out of 4 consecutive weights are set to zero.…
Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfe
Kiran Koshy Thekumparampil, Gaurush Hiranandani, Kousha Kalantari +2
We study learning of human preferences from a limited comparison feedback. This task is ubiquitous in machine learning. Its applications such as reinforcement learning from human f…
Learning the Target Network in Function Space
Kavosh Asadi, Yao Liu, Shoham Sabach +2
We focus on the task of learning the value function in the reinforcement learning (RL) setting. This task is often solved by updating a pair of online and target networks while ens…
MADA: Meta-Adaptive Optimizers through hyper-gradient Descent
Kaan Ozkara, Can Karakus, Parameswaran Raman +4
Following the introduction of Adam, several novel adaptive optimizers for deep learning have been proposed. These optimizers typically excel in some tasks but may not outperform Ad…
TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models
Zuxin Liu, Jesse Zhang, Kavosh Asadi +4
The full potential of large pretrained models remains largely untapped in control domains like robotics. This is mainly because of the scarcity of data and the computational challe…
Resetting the Optimizer in Deep RL: An Empirical Study
Kavosh Asadi, Rasool Fakoor, Shoham Sabach
We focus on the task of approximating the optimal value function in deep reinforcement learning. This iterative process is comprised of solving a sequence of optimization problems…
TD Convergence: An Optimization Perspective
Kavosh Asadi, Shoham Sabach, Yao Liu +2
We study the convergence behavior of the celebrated temporal-difference (TD) learning algorithm. By looking at the algorithm through the lens of optimization, we first argue that T…
Improved Complexities of Conditional Gradient-Type Methods with Applications to Robust Matrix Recovery Problems
Dan Garber, Shoham Sabach, Atara Kaplan
Motivated by robust matrix recovery problems such as Robust Principal Component Analysis, we consider a general optimization problem of minimizing a smooth and strongly convex loss…