Observation Space Matters: Benchmark and Optimization Algorithm
arXiv:2011.00756 · doi:10.1109/ICRA48506.2021.9561019
Abstract
Recent advances in deep reinforcement learning (deep RL) enable researchers to solve challenging control problems, from simulated environments to real-world robotic tasks. However, deep RL algorithms are known to be sensitive to the problem formulation, including observation spaces, action spaces, and reward functions. There exist numerous choices for observation spaces but they are often designed solely based on prior knowledge due to the lack of established principles. In this work, we conduct benchmark experiments to verify common design choices for observation spaces, such as Cartesian transformation, binary contact flags, a short history, or global positions. Then we propose a search algorithm to find the optimal observation spaces, which examines various candidate observation spaces and removes unnecessary observation channels with a Dropout-Permutation test. We demonstrate that our algorithm significantly improves learning speed compared to manually designed observation spaces. We also analyze the proposed algorithm by evaluating different hyperparameters.
References in corpus (23)
- Continuous control with deep reinforcement learning
- Neural Architecture Search with Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- Addressing Function Approximation Error in Actor-Critic Methods
- Learning agile and dynamic motor skills for legged robots
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Rethinking the Value of Network Pruning
- DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills
- SNIP: Single-shot Network Pruning based on Connection Sensitivity
- DeepMind Control Suite
- Deep Reinforcement Learning that Matters
- Variational Dropout Sparsifies Deep Neural Networks
- Uncertainty-Aware Reinforcement Learning for Collision Avoidance
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
- Visualizing and Understanding Atari Agents
- Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
- Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?
- What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study
- Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
- Learning to Walk in the Real World with Minimal Human Effort
- Learning to Locomote: Understanding How Environment Design Matters for Deep Reinforcement Learning
- Learning to Walk via Deep Reinforcement Learning