On the Expressivity of Neural Networks for Deep Reinforcement Learning

Abstract

We compare the model-free reinforcement learning with the model-based approaches through the lens of the expressive power of neural networks for policies, $Q$-functions, and dynamics. We show, theoretically and empirically, that even for one-dimensional continuous state space, there are many MDPs whose optimal $Q$-functions and policies are much more complex than the dynamics. We hypothesize many real-world MDPs also have a similar property. For these MDPs, model-based planning is a favorable algorithm, because the resulting policies can approximate the optimal policy significantly better than a neural network parameterization can, and model-free or model-based policy optimization rely on policy parameterization. Motivated by the theory, we apply a simple multi-step model-based bootstrapping planner (BOOTS) to bootstrap a weak $Q$-function into a stronger policy. Empirical results show that applying BOOTS on top of model-based or model-free policy optimization algorithms at the test time improves the performance on MuJoCo benchmark tasks.

Accepted in ICML 2020. Title of previous version was "Bootstrapping the Expressivity with Model-based Planning". Code is available at https://github.com/roosephu/boots

References in corpus (15)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Model-Based Reinforcement Learning for Atari
• Model-Ensemble Trust-Region Policy Optimization
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion
• Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees
• Fixup Initialization: Residual Learning Without Normalization
• Exploring Model-based Planning with Policy Networks
• Model-based RL in Contextual Decision Processes: PAC bounds and Exponential Improvements over Model-free Approaches
• Model-Augmented Actor-Critic: Backpropagating through Paths
• Diagnosing Bottlenecks in Deep Q-learning Algorithms
• Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?
• Calibrated Model-Based Deep Reinforcement Learning
• Task-Agnostic Dynamics Priors for Deep Reinforcement Learning