Harnessing Structures for Value-Based Planning and Reinforcement Learning

Abstract

Value-based methods constitute a fundamental methodology in planning and deep reinforcement learning (RL). In this paper, we propose to exploit the underlying structures of the state-action value function, i.e., Q function, for both planning and deep RL. In particular, if the underlying system dynamics lead to some global structures of the Q function, one should be capable of inferring the function better by leveraging such structures. Specifically, we investigate the low-rank structure, which widely exists for big data matrices. We verify empirically the existence of low-rank Q functions in the context of control and deep RL tasks. As our key contribution, by leveraging Matrix Estimation (ME) techniques, we propose a general framework to exploit the underlying low-rank structure in Q functions. This leads to a more efficient planning procedure for classical control, and additionally, a simple scheme that can be applied to any value-based RL techniques to consistently achieve better performance on "low-rank" tasks. Extensive experiments on control tasks and Atari games confirm the efficacy of our approach. Code is available at https://github.com/YyzHarry/SV-RL.

arXiv commentICLR 2020 (Oral)

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• DeepMind Control Suite
• Matrix estimation by Universal Singular Value Thresholding
• An overview of low-rank matrix recovery from incomplete observations
• Count-Based Exploration with Neural Density Models
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• Harnessing Structures in Big Data via Guaranteed Low-Rank Matrix Estimation
• Value function approximation via low-rank models

Cited by in corpus (6)

• On Robustness of Principal Component Regression
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• On Reinforcement Learning for Turn-based Zero-sum Markov Games
• Uncertainty-aware Low-Rank Q-Matrix Estimation for Deep Reinforcement Learning
• Non-Asymptotic Analysis of Monte Carlo Tree Search