Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research

Abstract

Since the introduction of DQN, a vast majority of reinforcement learning research has focused on reinforcement learning with deep neural networks as function approximators. New methods are typically evaluated on a set of environments that have now become standard, such as Atari 2600 games. While these benchmarks help standardize evaluation, their computational cost has the unfortunate side effect of widening the gap between those with ample access to computational resources, and those without. In this work we argue that, despite the community's emphasis on large-scale environments, the traditional small-scale environments can still yield valuable scientific insights and can help reduce the barriers to entry for underprivileged communities. To substantiate our claims, we empirically revisit the paper which introduced the Rainbow algorithm [Hessel et al., 2018] and present some new insights into the algorithms used by Rainbow.

arXiv commentProceedings of the 38th International Conference on Machine Learning (ICML 2021)

References in corpus (12)

• Dueling Network Architectures for Deep Reinforcement Learning
• Noisy Networks for Exploration
• Deep Reinforcement Learning that Matters
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Distributional Reinforcement Learning with Quantile Regression
• DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
• GEP-PG: Decoupling Exploration and Exploitation in Deep Reinforcement Learning Algorithms
• How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments
• MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments
• On Inductive Biases in Deep Reinforcement Learning
• A Comparative Analysis of Expected and Distributional Reinforcement Learning

Cited by in corpus (11)

• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• Reinforcement Learning-based Switching Controller for a Milliscale Robot in a Constrained Environment
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Distributional Reinforcement Learning with Unconstrained Monotonic Neural Networks
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• When does loss-based prioritization fail?
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning
• Large Batch Experience Replay
• On The Transferability of Deep-Q Networks