Reinforcement Learning via Fenchel-Rockafellar Duality

Abstract

We review basic concepts of convex duality, focusing on the very general and supremely useful Fenchel-Rockafellar duality. We summarize how this duality may be applied to a variety of reinforcement learning (RL) settings, including policy evaluation or optimization, online or offline learning, and discounted or undiscounted rewards. The derivations yield a number of intriguing results, including the ability to perform policy evaluation and on-policy policy gradient with behavior-agnostic offline data and methods to learn a policy via max-likelihood optimization. Although many of these results have appeared previously in various forms, we provide a unified treatment and perspective on these results, which we hope will enable researchers to better use and apply the tools of convex duality to make further progress in RL.

References in corpus (15)

• Maximum a Posteriori Policy Optimisation
• A unified view of entropy-regularized Markov decision processes
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
• Stochastic Variance Reduction Methods for Policy Evaluation
• GenDICE: Generalized Offline Estimation of Stationary Values
• Should one compute the Temporal Difference fix point or minimize the Bellman Residual? The unified oblique projection view
• Stochastic Primal-Dual Methods and Sample Complexity of Reinforcement Learning
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Imitation Learning as $f$-Divergence Minimization
• Scalable Bilinear $\pi$ Learning Using State and Action Features
• Randomized Linear Programming Solves the Discounted Markov Decision Problem In Nearly-Linear (Sometimes Sublinear) Running Time
• Exponential Family Estimation via Adversarial Dynamics Embedding
• f-Divergence constrained policy improvement
• Faster saddle-point optimization for solving large-scale Markov decision processes

Cited by in corpus (20)

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• MOReL : Model-Based Offline Reinforcement Learning
• Is Pessimism Provably Efficient for Offline RL?
• CoinDICE: Off-Policy Confidence Interval Estimation
• Single-Timescale Actor-Critic Provably Finds Globally Optimal Policy
• Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learning
• Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
• Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
• Statistical Bootstrapping for Uncertainty Estimation in Off-Policy Evaluation
• Offline Policy Selection under Uncertainty
• Twice regularized MDPs and the equivalence between robustness and regularization
• A Workflow for Offline Model-Free Robotic Reinforcement Learning
• Universal Off-Policy Evaluation
• The Information Geometry of Unsupervised Reinforcement Learning
• Off-policy Reinforcement Learning with Optimistic Exploration and Distribution Correction
• Supervised Off-Policy Ranking
• Convex Regularization in Monte-Carlo Tree Search
• Near Optimal Policy Optimization via REPS
• A maximum-entropy approach to off-policy evaluation in average-reward MDPs
• Distributionally-Constrained Policy Optimization via Unbalanced Optimal Transport