Ranking Policy Gradient

Abstract

Sample inefficiency is a long-lasting problem in reinforcement learning (RL). The state-of-the-art estimates the optimal action values while it usually involves an extensive search over the state-action space and unstable optimization. Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy gradient method that learns the optimal rank of a set of discrete actions. To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles. These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sample-efficiency. Furthermore, the sample complexity of RPG does not depend on the dimension of state space, which enables RPG for large-scale problems. We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.

References in corpus (19)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• Deep Q-learning from Demonstrations
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Bridging the Gap Between Value and Policy Based Reinforcement Learning
• Sample Efficient Actor-Critic with Experience Replay
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation
• Reinforcement and Imitation Learning via Interactive No-Regret Learning
• Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
• Combining policy gradient and Q-learning
• Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
• Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Bounds
• If MaxEnt RL is the Answer, What is the Question?
• Convergent Tree Backup and Retrace with Function Approximation
• Variational Bayesian Reinforcement Learning with Regret Bounds