Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation

Abstract

In this work, we propose to apply trust region optimization to deep reinforcement learning using a recently proposed Kronecker-factored approximation to the curvature. We extend the framework of natural policy gradient and propose to optimize both the actor and the critic using Kronecker-factored approximate curvature (K-FAC) with trust region; hence we call our method Actor Critic using Kronecker-Factored Trust Region (ACKTR). To the best of our knowledge, this is the first scalable trust region natural gradient method for actor-critic methods. It is also a method that learns non-trivial tasks in continuous control as well as discrete control policies directly from raw pixel inputs. We tested our approach across discrete domains in Atari games as well as continuous domains in the MuJoCo environment. With the proposed methods, we are able to achieve higher rewards and a 2- to 3-fold improvement in sample efficiency on average, compared to previous state-of-the-art on-policy actor-critic methods. Code is available at https://github.com/openai/baselines

14 pages, 9 figures; update github repo link

References in corpus (2)

• Emergence of Locomotion Behaviours in Rich Environments
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature

Cited by in corpus (139)

• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• Deep Reinforcement Learning: An Overview
• Soft Actor-Critic for Discrete Action Settings
• Simple random search provides a competitive approach to reinforcement learning
• Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review
• Multi-agent Reinforcement Learning for Cooperative Lane Changing of Connected and Autonomous Vehicles in Mixed Traffic
• A Survey of Deep Reinforcement Learning in Video Games
• Tianshou: a Highly Modularized Deep Reinforcement Learning Library
• Reinforcement Learning Algorithms: An Overview and Classification
• A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Neural Policy Gradient Methods: Global Optimality and Rates of Convergence
• Learning Montezuma's Revenge from a Single Demonstration
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• Queueing Network Controls via Deep Reinforcement Learning
• Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Unsupervised Video Object Segmentation for Deep Reinforcement Learning
• First Order Constrained Optimization in Policy Space
• Phasic Policy Gradient
• A reinforcement learning approach to rare trajectory sampling
• Dealing with Sparse Rewards in Reinforcement Learning
• AllenAct: A Framework for Embodied AI Research
• A Survey of Optimization Methods from a Machine Learning Perspective
• Deep Multi-agent Reinforcement Learning for Highway On-Ramp Merging in Mixed Traffic
• Truly Proximal Policy Optimization
• Natural Environment Benchmarks for Reinforcement Learning
• Diff-DAC: Distributed Actor-Critic for Average Multitask Deep Reinforcement Learning
• Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems
• An Adaptive Clipping Approach for Proximal Policy Optimization
• Noisy Natural Gradient as Variational Inference
• Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives
• A Tour of Reinforcement Learning: The View from Continuous Control
• SquirRL: Automating Attack Analysis on Blockchain Incentive Mechanisms with Deep Reinforcement Learning
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• A Machine Learning Approach to Routing
• Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems
• Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments
• Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL
• Vulnerability-Aware Poisoning Mechanism for Online RL with Unknown Dynamics
• Discretizing Continuous Action Space for On-Policy Optimization
• Obtaining Robust Control and Navigation Policies for Multi-Robot Navigation via Deep Reinforcement Learning
• Online 3D Bin Packing with Constrained Deep Reinforcement Learning
• Applications of Multi-Agent Reinforcement Learning in Future Internet: A Comprehensive Survey
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game
• Action Semantics Network: Considering the Effects of Actions in Multiagent Systems
• Adversarially Guided Actor-Critic
• Intelligent Trading Systems: A Sentiment-Aware Reinforcement Learning Approach
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• ConfuciuX: Autonomous Hardware Resource Assignment for DNN Accelerators using Reinforcement Learning
• Options as responses: Grounding behavioural hierarchies in multi-agent RL
• Kalman meets Bellman: Improving Policy Evaluation through Value Tracking
• Modularity in Deep Learning: A Survey
• Maximum Mutation Reinforcement Learning for Scalable Control
• Towards Similarity Graphs Constructed by Deep Reinforcement Learning
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Kronecker Recurrent Units
• Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents
• Effective Exploration for Deep Reinforcement Learning via Bootstrapped Q-Ensembles under Tsallis Entropy Regularization
• ROS2Learn: a reinforcement learning framework for ROS 2
• A Coordinate-Free Construction of Scalable Natural Gradient
• Entanglement engineering of optomechanical systems by reinforcement learning
• Fourier Policy Gradients
• Scalable Multi-Agent Inverse Reinforcement Learning via Actor-Attention-Critic
• Solving the scalarization issues of Advantage-based Reinforcement Learning Algorithms
• A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learning
• Policy Optimization With Penalized Point Probability Distance: An Alternative To Proximal Policy Optimization
• Accelerating Natural Gradient with Higher-Order Invariance
• Learning Practically Feasible Policies for Online 3D Bin Packing
• Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning
• Partially Detected Intelligent Traffic Signal Control: Environmental Adaptation
• Expert-augmented actor-critic for ViZDoom and Montezumas Revenge
• Unsupervised Reinforcement Learning of Transferable Meta-Skills for Embodied Navigation
• Structured Control Nets for Deep Reinforcement Learning
• Robust Imitation Learning from Noisy Demonstrations
• A Visual Communication Map for Multi-Agent Deep Reinforcement Learning
• Natural Gradient Deep Q-learning
• Off-Policy Actor-Critic in an Ensemble: Achieving Maximum General Entropy and Effective Environment Exploration in Deep Reinforcement Learning
• Is the Policy Gradient a Gradient?
• Only Relevant Information Matters: Filtering Out Noisy Samples to Boost RL
• KoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge
• Trust Region Value Optimization using Kalman Filtering
• PPMC RL Training Algorithm: Rough Terrain Intelligent Robots through Reinforcement Learning
• Reinforcement Learning with Structured Hierarchical Grammar Representations of Actions
• RL STaR Platform: Reinforcement Learning for Simulation based Training of Robots
• The Adversarial Resilience Learning Architecture for AI-based Modelling, Exploration, and Operation of Complex Cyber-Physical Systems
• Stealing Deep Reinforcement Learning Models for Fun and Profit
• On-Policy Trust Region Policy Optimisation with Replay Buffers
• Hierarchical Learning for Modular Robots
• On the Search for Feedback in Reinforcement Learning
• LeRoP: A Learning-Based Modular Robot Photography Framework
• Policy Search by Target Distribution Learning for Continuous Control
• Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning
• Disentangling Dynamics and Returns: Value Function Decomposition with Future Prediction
• Deep reinforced learning enables solving rich discrete-choice life cycle models to analyze social security reforms
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Network Automatic Pruning: Start NAP and Take a Nap
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• An Empirical Analysis of Proximal Policy Optimization with Kronecker-factored Natural Gradients
• Scalable and Practical Natural Gradient for Large-Scale Deep Learning
• Recurrent Value Functions
• Monte-Carlo Tree Search for Policy Optimization
• D2C 2.0: Decoupled Data-Based Approach for Learning to Control Stochastic Nonlinear Systems via Model-Free ILQR
• NavigationNet: A Large-scale Interactive Indoor Navigation Dataset
• Average-Reward Reinforcement Learning with Trust Region Methods
• Deep Learned Path Planning via Randomized Reward-Linked-Goals and Potential Space Applications
• Biologically inspired architectures for sample-efficient deep reinforcement learning
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• Reinforcement Learning for Nested Polar Code Construction
• Minimalistic Attacks: How Little it Takes to Fool a Deep Reinforcement Learning Policy
• Genetic-Gated Networks for Deep Reinforcement
• Learning Policies through Quantile Regression
• Hindsight Experience Replay with Kronecker Product Approximate Curvature
• Quasi-Newton Trust Region Policy Optimization
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Which Channel to Ask My Question? Personalized Customer Service Request Stream Routing using Deep Reinforcement Learning
• Rogue-Gym: A New Challenge for Generalization in Reinforcement Learning
• Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
• Regularly Updated Deterministic Policy Gradient Algorithm
• Guide Actor-Critic for Continuous Control
• Perturbation-based exploration methods in deep reinforcement learning
• Weighted Entropy Modification for Soft Actor-Critic
• Improving On-policy Learning with Statistical Reward Accumulation
• Learning Robust and Adaptive Real-World Continuous Control Using Simulation and Transfer Learning
• Emergence of Different Modes of Tool Use in a Reaching and Dragging Task
• Reinforcement Learning for Robust Missile Autopilot Design
• Deep Active Localization
• FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control
• Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning
• Hard Attention Control By Mutual Information Maximization
• Conditioning of Reinforcement Learning Agents and its Policy Regularization Application
• Robust Implicit Backpropagation
• Mix and Mask Actor-Critic Methods
• Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents
• DROMO: Distributionally Robust Offline Model-based Policy Optimization
• Low-Variance Policy Gradient Estimation with World Models
• Learning to Transcribe by Ear