MDP Playground: An Analysis and Debug Testbed for Reinforcement Learning

Abstract

We present MDP Playground, a testbed for Reinforcement Learning (RL) agents with dimensions of hardness that can be controlled independently to challenge agents in different ways and obtain varying degrees of hardness in toy and complex RL environments. We consider and allow control over a wide variety of dimensions, including delayed rewards, sequence lengths, reward density, stochasticity, image representations, irrelevant features, time unit, action range and more. We define a parameterised collection of fast-to-run toy environments in OpenAI Gym by varying these dimensions and propose to use these to understand agents better. We then show how to design experiments using MDP Playground to gain insights on the toy environments. We also provide wrappers that can inject many of these dimensions into any Gym environment. We experiment with these wrappers on Atari and Mujoco to allow for understanding the effects of these dimensions on environments that are more complex than the toy environments. We also compare the effect of the dimensions on the toy and complex environments. Finally, we show how to use MDP Playground to debug agents, to study the interaction of multiple dimensions and describe further use-cases.

arXiv commentSame version as the one published in JAIR Vol. 77 (2023)

References in corpus (24)

• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Asynchronous Methods for Deep Reinforcement Learning
• Understanding deep learning requires rethinking generalization
• Noisy Networks for Exploration
• RLlib: Abstractions for Distributed Reinforcement Learning
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Benchmarking Model-Based Reinforcement Learning
• Meta-Gradient Reinforcement Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• RUDDER: Return Decomposition for Delayed Rewards
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping
• How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategies
• Environment-Independent Task Specifications via GLTL
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• MNIST-C: A Robustness Benchmark for Computer Vision
• On the Importance of Hyperparameter Optimization for Model-based Reinforcement Learning
• Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learning
• On Learning Intrinsic Rewards for Policy Gradient Methods
• Self-Paced Contextual Reinforcement Learning
• TempoRL: Learning When to Act
• Reward Tweaking: Maximizing the Total Reward While Planning for Short Horizons