Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning

Abstract

Recently many algorithms were devised for reinforcement learning (RL) with function approximation. While they have clear algorithmic distinctions, they also have many implementation differences that are algorithm-independent and sometimes under-emphasized. Such mixing of algorithmic novelty and implementation craftsmanship makes rigorous analyses of the sources of performance improvements across algorithms difficult. In this work, we focus on a series of off-policy inference-based actor-critic algorithms -- MPO, AWR, and SAC -- to decouple their algorithmic innovations and implementation decisions. We present unified derivations through a single control-as-inference objective, where we can categorize each algorithm as based on either Expectation-Maximization (EM) or direct Kullback-Leibler (KL) divergence minimization and treat the rest of specifications as implementation details. We performed extensive ablation studies, and identified substantial performance drops whenever implementation details are mismatched for algorithmic choices. These results show which implementation or code details are co-adapted and co-evolved with algorithms, and which are transferable across algorithms: as examples, we identified that tanh Gaussian policy and network sizes are highly adapted to algorithmic types, while layer normalization and ELU are critical for MPO's performances but also transfer to noticeable gains in SAC. We hope our work can inspire future work to further demystify sources of performance improvements across multiple algorithms and allow researchers to build on one another's both algorithmic and implementational innovations.

arXiv commentAccepted at NeurIPS 2021. The implementation is available at: https://github.com/frt03/inference-based-rl

References in corpus (34)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Soft Actor-Critic Algorithms and Applications
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Reinforcement Learning with Deep Energy-Based Policies
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• Deep Reinforcement Learning that Matters
• Continuous Deep Q-Learning with Model-based Acceleration
• Benchmarking Model-Based Reinforcement Learning
• Maximum a Posteriori Policy Optimisation
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• Self-Imitation Learning
• Critic Regularized Regression
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• A Theory of Regularized Markov Decision Processes
• Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
• ChainerRL: A Deep Reinforcement Learning Library
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• Acme: A Research Framework for Distributed Reinforcement Learning
• Relative Entropy Regularized Policy Iteration
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Variational Inference MPC for Bayesian Model-based Reinforcement Learning
• Expected Policy Gradients for Reinforcement Learning
• VIREL: A Variational Inference Framework for Reinforcement Learning
• Action and Perception as Divergence Minimization
• Action-depedent Control Variates for Policy Optimization via Stein's Identity
• Making Sense of Reinforcement Learning and Probabilistic Inference
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• Improving Policy Gradient by Exploring Under-appreciated Rewards