Direct and indirect reinforcement learning
arXiv:1912.10600
Abstract
Reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. In this paper, we classify RL into direct and indirect RL according to how they seek the optimal policy of the Markov decision process problem. The former solves the optimal policy by directly maximizing an objective function using gradient descent methods, in which the objective function is usually the expectation of accumulative future rewards. The latter indirectly finds the optimal policy by solving the Bellman equation, which is the sufficient and necessary condition from Bellman's principle of optimality. We study policy gradient forms of direct and indirect RL and show that both of them can derive the actor-critic architecture and can be unified into a policy gradient with the approximate value function and the stationary state distribution, revealing the equivalence of direct and indirect RL. We employ a Gridworld task to verify the influence of different forms of policy gradient, suggesting their differences and relationships experimentally. Finally, we classify current mainstream RL algorithms using the direct and indirect taxonomy, together with other ones including value-based and policy-based, model-based and model-free.
arXiv commentPublished in International Journal of Intelligent Systems
References in corpus (16)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Prioritized Experience Replay
- Dueling Network Architectures for Deep Reinforcement Learning
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Distributed Prioritized Experience Replay
- Recurrent World Models Facilitate Policy Evolution
- Model-Ensemble Trust-Region Policy Optimization
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Sample Efficient Actor-Critic with Experience Replay
- Equivalence Between Policy Gradients and Soft Q-Learning
- Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning
- Combining policy gradient and Q-learning
- Trust-PCL: An Off-Policy Trust Region Method for Continuous Control
- Generalized Off-Policy Actor-Critic