Distributed Distributional Deterministic Policy Gradients

Abstract

This work adopts the very successful distributional perspective on reinforcement learning and adapts it to the continuous control setting. We combine this within a distributed framework for off-policy learning in order to develop what we call the Distributed Distributional Deep Deterministic Policy Gradient algorithm, D4PG. We also combine this technique with a number of additional, simple improvements such as the use of $N$-step returns and prioritized experience replay. Experimentally we examine the contribution of each of these individual components, and show how they interact, as well as their combined contributions. Our results show that across a wide variety of simple control tasks, difficult manipulation tasks, and a set of hard obstacle-based locomotion tasks the D4PG algorithm achieves state of the art performance.

References in corpus (4)

• Continuous control with deep reinforcement learning
• Emergence of Locomotion Behaviours in Rich Environments
• Distributed Prioritized Experience Replay
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation

Cited by in corpus (95)

• Deep Reinforcement Learning for Cyber Security
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Learning Dexterous In-Hand Manipulation
• Federated Reinforcement Learning: Techniques, Applications, and Open Challenges
• Autonomous Unmanned Aerial Vehicle Navigation using Reinforcement Learning: A Systematic Review
• Dream to Control: Learning Behaviors by Latent Imagination
• Reinforcement Learning Algorithms: An Overview and Classification
• Review: Deep Learning in Electron Microscopy
• Never Give Up: Learning Directed Exploration Strategies
• Deep Reinforcement Learning for Radio Resource Allocation and Management in Next Generation Heterogeneous Wireless Networks: A Survey
• Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints
• Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model
• Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving
• Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• An empirical investigation of the challenges of real-world reinforcement learning
• Softmax Deep Double Deterministic Policy Gradients
• Scaling data-driven robotics with reward sketching and batch reinforcement learning
• Learning Gentle Object Manipulation with Curiosity-Driven Deep Reinforcement Learning
• Longitudinal Dynamic versus Kinematic Models for Car-Following Control Using Deep Reinforcement Learning
• Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures
• Primal Wasserstein Imitation Learning
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Learning Purely Tactile In-Hand Manipulation with a Torque-Controlled Hand
• Worst Cases Policy Gradients
• Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatment
• Robust MAML: Prioritization task buffer with adaptive learning process for model-agnostic meta-learning
• Task-Relevant Adversarial Imitation Learning
• Sparse Graphical Memory for Robust Planning
• What Matters for Adversarial Imitation Learning?
• Dynamics-aware Embeddings
• Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods
• Online 3D Bin Packing with Constrained Deep Reinforcement Learning
• Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
• Time Reversal as Self-Supervision
• Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
• GAN-powered Deep Distributional Reinforcement Learning for Resource Management in Network Slicing
• SpikePropamine: Differentiable Plasticity in Spiking Neural Networks
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• Positive-Unlabeled Reward Learning
• Dynamic Experience Replay
• Meta-Learning surrogate models for sequential decision making
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization
• AndroidEnv: A Reinforcement Learning Platform for Android
• Likelihood Quantile Networks for Coordinating Multi-Agent Reinforcement Learning
• Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs
• Deep Reinforcement Learning with Decorrelation
• Tonic: A Deep Reinforcement Learning Library for Fast Prototyping and Benchmarking
• Contrastive Variational Reinforcement Learning for Complex Observations
• Online Hyper-parameter Tuning in Off-policy Learning via Evolutionary Strategies
• Balancing Constraints and Rewards with Meta-Gradient D4PG
• Learning Cooperative Multi-Agent Policies with Partial Reward Decoupling
• Learning Practically Feasible Policies for Online 3D Bin Packing
• Bridging Imagination and Reality for Model-Based Deep Reinforcement Learning
• Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
• Implicit Distributional Reinforcement Learning
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration
• Model-free and Bayesian Ensembling Model-based Deep Reinforcement Learning for Particle Accelerator Control Demonstrated on the FERMI FEL
• Distributional Actor-Critic Ensemble for Uncertainty-Aware Continuous Control
• FIXAR: A Fixed-Point Deep Reinforcement Learning Platform with Quantization-Aware Training and Adaptive Parallelism
• Task-Agnostic Morphology Evolution
• Robust Constrained Reinforcement Learning for Continuous Control with Model Misspecification
• Learning Performance Graphs from Demonstrations via Task-Based Evaluations
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Towards Automatic Actor-Critic Solutions to Continuous Control
• On Instrumental Variable Regression for Deep Offline Policy Evaluation
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• Distributed Soft Actor-Critic with Multivariate Reward Representation and Knowledge Distillation
• Composite Q-learning: Multi-scale Q-function Decomposition and Separable Optimization
• Learning to Solve a Rubik's Cube with a Dexterous Hand
• Distributed Uplink Beamforming in Cell-Free Networks Using Deep Reinforcement Learning
• ACE: An Actor Ensemble Algorithm for Continuous Control with Tree Search
• Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning
• Stochastically Dominant Distributional Reinforcement Learning
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Reinforcement Learning for Vision-based Object Manipulation with Non-parametric Policy and Action Primitives
• Pessimistic Model Selection for Offline Deep Reinforcement Learning
• Attention-based Deep Reinforcement Learning for Multi-view Environments
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• A Decoupled Data Based Approach to Stochastic Optimal Control Problems
• Lineage Evolution Reinforcement Learning
• Reinforcement Learning for Robust Missile Autopilot Design
• GMAC: A Distributional Perspective on Actor-Critic Framework
• Learning to Reach, Swim, Walk and Fly in One Trial: Data-Driven Control with Scarce Data and Side Information
• Active Offline Policy Selection
• A Policy Efficient Reduction Approach to Convex Constrained Deep Reinforcement Learning
• Biased Estimates of Advantages over Path Ensembles
• IGN : Implicit Generative Networks
• Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Space
• A Study of Policy Gradient on a Class of Exactly Solvable Models
• LBGP: Learning Based Goal Planning for Autonomous Following in Front
• Generative Adversarial Imagination for Sample Efficient Deep Reinforcement Learning