Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research

Abstract

The purpose of this technical report is two-fold. First of all, it introduces a suite of challenging continuous control tasks (integrated with OpenAI Gym) based on currently existing robotics hardware. The tasks include pushing, sliding and pick & place with a Fetch robotic arm as well as in-hand object manipulation with a Shadow Dexterous Hand. All tasks have sparse binary rewards and follow a Multi-Goal Reinforcement Learning (RL) framework in which an agent is told what to do using an additional input. The second part of the paper presents a set of concrete research ideas for improving RL algorithms, most of which are related to Multi-Goal RL and Hindsight Experience Replay.

References in corpus (7)

• Continuous control with deep reinforcement learning
• Parameter Space Noise for Exploration
• Equivalence Between Policy Gradients and Soft Q-Learning
• Temporal Difference Models: Model-Free Deep RL for Model-Based Control
• Reverse Curriculum Generation for Reinforcement Learning
• Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
• Hindsight policy gradients

Cited by in corpus (95)

• Solving Rubik's Cube with a Robot Hand
• QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning
• First return, then explore
• Efficient Exploration via State Marginal Matching
• Variational Option Discovery Algorithms
• Open-Sourced Reinforcement Learning Environments for Surgical Robotics
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration
• Accelerating Reinforcement Learning for Reaching using Continuous Curriculum Learning
• Residual Policy Learning
• InfoBot: Transfer and Exploration via the Information Bottleneck
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Curiosity-Driven Experience Prioritization via Density Estimation
• Reinforcement Learning for Robotic Manipulation using Simulated Locomotion Demonstrations
• VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
• Mapping State Space using Landmarks for Universal Goal Reaching
• Natural Environment Benchmarks for Reinforcement Learning
• Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning
• World Model as a Graph: Learning Latent Landmarks for Planning
• Analysing Deep Reinforcement Learning Agents Trained with Domain Randomisation
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Generalized Hindsight for Reinforcement Learning
• Training Agents using Upside-Down Reinforcement Learning
• Grounding Language to Autonomously-Acquired Skills via Goal Generation
• Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
• Model-based Reinforcement Learning from Signal Temporal Logic Specifications
• ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots
• Curiosity-Driven Multi-Criteria Hindsight Experience Replay
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Universal Successor Features for Transfer Reinforcement Learning
• Soft Hindsight Experience Replay
• Disentangled Skill Embeddings for Reinforcement Learning
• Directed Exploration for Reinforcement Learning
• CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents
• Causal Influence Detection for Improving Efficiency in Reinforcement Learning
• Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks
• Measuring Visual Generalization in Continuous Control from Pixels
• Circus ANYmal: A Quadruped Learning Dexterous Manipulation with Its Limbs
• Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling
• Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learning
• Information Theoretic Model Predictive Q-Learning
• TAAC: Temporally Abstract Actor-Critic for Continuous Control
• Complementary reinforcement learning towards explainable agents
• Hindsight Generative Adversarial Imitation Learning
• Language-Conditioned Goal Generation: a New Approach to Language Grounding for RL
• How Much Do Unstated Problem Constraints Limit Deep Robotic Reinforcement Learning?
• Discovery of Options via Meta-Learned Subgoals
• A User's Guide to Calibrating Robotics Simulators
• An Inductive Bias for Distances: Neural Nets that Respect the Triangle Inequality
• Deep Learning with Experience Ranking Convolutional Neural Network for Robot Manipulator
• Solving Challenging Dexterous Manipulation Tasks With Trajectory Optimisation and Reinforcement Learning
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• Mutual Information State Intrinsic Control
• Competitive Experience Replay
• Goal-Directed Planning by Reinforcement Learning and Active Inference
• Exploring Restart Distributions
• Learning Powerful Policies by Using Consistent Dynamics Model
• Towards More Sample Efficiency in Reinforcement Learning with Data Augmentation
• Diversity-based Trajectory and Goal Selection with Hindsight Experience Replay
• Exploration via Hindsight Goal Generation
• Learning Parametric Constraints in High Dimensions from Demonstrations
• Complex Robotic Manipulation via Graph-Based Hindsight Goal Generation
• Unbiased Methods for Multi-Goal Reinforcement Learning
• Imaginary Hindsight Experience Replay: Curious Model-based Learning for Sparse Reward Tasks
• Adversarial Active Exploration for Inverse Dynamics Model Learning
• Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning
• Follow the Object: Curriculum Learning for Manipulation Tasks with Imagined Goals
• Novel Policy Seeking with Constrained Optimization
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning
• MGHRL: Meta Goal-generation for Hierarchical Reinforcement Learning
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• Effects of sparse rewards of different magnitudes in the speed of learning of model-based actor critic methods
• Subgoal-based Reward Shaping to Improve Efficiency in Reinforcement Learning
• Self-Adapting Recurrent Models for Object Pushing from Learning in Simulation
• Multi-task Reinforcement Learning with a Planning Quasi-Metric
• Warm Starting of CMA-ES for Contextual Optimization Problems
• A Generalised Inverse Reinforcement Learning Framework
• Improvements on Hindsight Learning
• Hindsight Experience Replay with Kronecker Product Approximate Curvature
• Episodic Self-Imitation Learning with Hindsight
• C-Learning: Horizon-Aware Cumulative Accessibility Estimation
• Learning Provably Robust Motion Planners Using Funnel Libraries
• Quickest change detection with unknown parameters: Constant complexity and near optimality
• Evolutionary Stochastic Policy Distillation
• Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding
• Reward Shaping with Dynamic Trajectory Aggregation
• Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard
• Generative Actor-Critic: An Off-policy Algorithm Using the Push-forward Model
• Deep Reinforcement Learning for Complex Manipulation Tasks with Sparse Feedback
• Zero-shot Policy Learning with Spatial Temporal RewardDecomposition on Contingency-aware Observation
• BlockPuzzle - A Challenge in Physical Reasoning and Generalization for Robot Learning
• A Logarithmic Barrier Method For Proximal Policy Optimization
• Solving the Real Robot Challenge using Deep Reinforcement Learning