Rainbow: Combining Improvements in Deep Reinforcement Learning

Abstract

The deep reinforcement learning community has made several independent improvements to the DQN algorithm. However, it is unclear which of these extensions are complementary and can be fruitfully combined. This paper examines six extensions to the DQN algorithm and empirically studies their combination. Our experiments show that the combination provides state-of-the-art performance on the Atari 2600 benchmark, both in terms of data efficiency and final performance. We also provide results from a detailed ablation study that shows the contribution of each component to overall performance.

Under review as a conference paper at AAAI 2018

Cited by in corpus (220)

• Bootstrap your own latent: A new approach to self-supervised Learning
• A Survey and Critique of Multiagent Deep Reinforcement Learning
• A Review of Deep Reinforcement Learning for Smart Building Energy Management
• Implicit Quantile Networks for Distributional Reinforcement Learning
• Collision Avoidance in Pedestrian-Rich Environments with Deep Reinforcement Learning
• Go-Explore: a New Approach for Hard-Exploration Problems
• Wi-Fi Meets ML: A Survey on Improving IEEE 802.11 Performance with Machine Learning
• Leveraging Procedural Generation to Benchmark Reinforcement Learning
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Dream to Control: Learning Behaviors by Latent Imagination
• Path-Restore: Learning Network Path Selection for Image Restoration
• Unsupervised Learning of Object Keypoints for Perception and Control
• AlgaeDICE: Policy Gradient from Arbitrary Experience
• ChainerRL: A Deep Reinforcement Learning Library
• Decoupling Representation Learning from Reinforcement Learning
• DRLE: Decentralized Reinforcement Learning at the Edge for Traffic Light Control in the IoV
• Discovering Reinforcement Learning Algorithms
• Queueing Network Controls via Deep Reinforcement Learning
• Deep reinforcement learning for guidewire navigation in coronary artery phantom
• Hyperbolic Discounting and Learning over Multiple Horizons
• Learning Dynamic Belief Graphs to Generalize on Text-Based Games
• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics
• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization
• Data-Efficient Reinforcement Learning with Self-Predictive Representations
• Data-driven control of micro-climate in buildings: an event-triggered reinforcement learning approach
• Improving Generalization in Reinforcement Learning with Mixture Regularization
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning
• The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
• d3rlpy: An Offline Deep Reinforcement Learning Library
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Measuring the Reliability of Reinforcement Learning Algorithms
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past
• Ray Interference: a Source of Plateaus in Deep Reinforcement Learning
• Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation
• Meta-Gradient Reinforcement Learning with an Objective Discovered Online
• Hyp-RL : Hyperparameter Optimization by Reinforcement Learning
• DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction
• Pretraining Representations for Data-Efficient Reinforcement Learning
• Prioritized Sequence Experience Replay
• Qualitative Measurements of Policy Discrepancy for Return-Based Deep Q-Network
• Online and Offline Reinforcement Learning by Planning with a Learned Model
• Online Deep Reinforcement Learning for Autonomous UAV Navigation and Exploration of Outdoor Environments
• Mastering Atari with Discrete World Models
• Benchmarking Bonus-Based Exploration Methods on the Arcade Learning Environment
• State Entropy Maximization with Random Encoders for Efficient Exploration
• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning
• A Comparative Analysis of Expected and Distributional Reinforcement Learning
• On Bonus-Based Exploration Methods in the Arcade Learning Environment
• Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field
• SAMBA: Safe Model-Based & Active Reinforcement Learning
• Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games
• Finite-Time Analysis for Double Q-learning
• CopyCAT: Taking Control of Neural Policies with Constant Attacks
• Experience Replay Optimization
• Modern Deep Reinforcement Learning Algorithms
• Learning to Navigate in Indoor Environments: from Memorizing to Reasoning
• Obstacle Avoidance Using a Monocular Camera
• LaProp: Separating Momentum and Adaptivity in Adam
• Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset
• Obtaining Robust Control and Navigation Policies for Multi-Robot Navigation via Deep Reinforcement Learning
• Deep Bayesian Reward Learning from Preferences
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data
• Optimistic Exploration even with a Pessimistic Initialisation
• On the Estimation Bias in Double Q-Learning
• Hedging using reinforcement learning: Contextual $k$-Armed Bandit versus $Q$-learning
• Striving for Simplicity and Performance in Off-Policy DRL: Output Normalization and Non-Uniform Sampling
• Learn to Interpret Atari Agents
• An initial attempt of combining visual selective attention with deep reinforcement learning
• Online Robustness Training for Deep Reinforcement Learning
• Synthetic Returns for Long-Term Credit Assignment
• World Discovery Models
• Simplifying Deep Reinforcement Learning via Self-Supervision
• Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspective
• Muesli: Combining Improvements in Policy Optimization
• Behavior From the Void: Unsupervised Active Pre-Training
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• Student-Initiated Action Advising via Advice Novelty
• Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning
• Simultaneous Navigation and Radio Mapping for Cellular-Connected UAV with Deep Reinforcement Learning
• Training Larger Networks for Deep Reinforcement Learning
• Interference and Generalization in Temporal Difference Learning
• OctoPath: An OcTree Based Self-Supervised Learning Approach to Local Trajectory Planning for Mobile Robots
• Schedule Earth Observation satellites with Deep Reinforcement Learning
• DREAM: Deep Regret minimization with Advantage baselines and Model-free learning
• Revisiting Parameter Sharing in Multi-Agent Deep Reinforcement Learning
• General non-linear Bellman equations
• DOM-Q-NET: Grounded RL on Structured Language
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Adversarial Attacks on Reinforcement Learning based Energy Management Systems of Extended Range Electric Delivery Vehicles
• Affordance-based Reinforcement Learning for Urban Driving
• Improving Generative Imagination in Object-Centric World Models
• Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations
• The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learning
• OffWorld Gym: open-access physical robotics environment for real-world reinforcement learning benchmark and research
• Retrospective Analysis of the 2019 MineRL Competition on Sample Efficient Reinforcement Learning
• MULEX: Disentangling Exploitation from Exploration in Deep RL
• Adapting Behaviour for Learning Progress
• Self-Imitation Advantage Learning
• PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning
• AgentGraph: Towards Universal Dialogue Management with Structured Deep Reinforcement Learning
• Regret Bounds for Discounted MDPs
• E-commerce warehousing: learning a storage policy
• Scaling Imitation Learning in Minecraft
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• Deep Reinforcement Learning with Feedback-based Exploration
• MineRL: A Large-Scale Dataset of Minecraft Demonstrations
• Convergent and Efficient Deep Q Network Algorithm
• Automated Lane Change Decision Making using Deep Reinforcement Learning in Dynamic and Uncertain Highway Environment
• Auto-MAP: A DQN Framework for Exploring Distributed Execution Plans for DNN Workloads
• Multiplayer Support for the Arcade Learning Environment
• Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments
• A learning gap between neuroscience and reinforcement learning
• Exploration of Reinforcement Learning for Event Camera using Car-like Robots
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• The Adversarial Resilience Learning Architecture for AI-based Modelling, Exploration, and Operation of Complex Cyber-Physical Systems
• Variance Reduction for Deep Q-Learning using Stochastic Recursive Gradient
• Theory of Mind for Deep Reinforcement Learning in Hanabi
• Deep Reinforcement Learning for Optimal Stopping with Application in Financial Engineering
• Evaluating the Rainbow DQN Agent in Hanabi with Unseen Partners
• Recommendation Fairness: From Static to Dynamic
• TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL
• WAD: A Deep Reinforcement Learning Agent for Urban Autonomous Driving
• Planning from Pixels using Inverse Dynamics Models
• Mixture of Step Returns in Bootstrapped DQN
• Hierarchically Integrated Models: Learning to Navigate from Heterogeneous Robots
• Revisiting Prioritized Experience Replay: A Value Perspective
• NeurIPS 2021 Competition IGLU: Interactive Grounded Language Understanding in a Collaborative Environment
• An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learning
• Join Query Optimization with Deep Reinforcement Learning Algorithms
• Correlation-aware Cooperative Multigroup Broadcast 360° Video Delivery Network: A Hierarchical Deep Reinforcement Learning Approach
• Adaptive Traffic Control with Deep Reinforcement Learning: Towards State-of-the-art and Beyond
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• Deep Reinforcement Learning for Backscatter-Aided Data Offloading in Mobile Edge Computing
• Finding Needles in a Moving Haystack: Prioritizing Alerts with Adversarial Reinforcement Learning
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• QVMix and QVMix-Max: Extending the Deep Quality-Value Family of Algorithms to Cooperative Multi-Agent Reinforcement Learning
• Semantic RL with Action Grammars: Data-Efficient Learning of Hierarchical Task Abstractions
• Mitigating Planner Overfitting in Model-Based Reinforcement Learning
• Temporally-Extended ε-Greedy Exploration
• Learning to Sample with Local and Global Contexts in Experience Replay Buffer
• Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning
• Reinforcement Learning for Industrial Control Network Cyber Security Orchestration
• Optimal Status Update for Caching Enabled IoT Networks: A Dueling Deep R-Network Approach
• Deep Model-Based Reinforcement Learning via Estimated Uncertainty and Conservative Policy Optimization
• Temporal-Difference Value Estimation via Uncertainty-Guided Soft Updates
• Explainable Deep Reinforcement Learning Using Introspection in a Non-episodic Task
• Approximating two value functions instead of one: towards characterizing a new family of Deep Reinforcement Learning algorithms
• Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning
• Bias-reduced Multi-step Hindsight Experience Replay for Efficient Multi-goal Reinforcement Learning
• Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience
• TDprop: Does Jacobi Preconditioning Help Temporal Difference Learning?
• Model-Free Episodic Control with State Aggregation
• Learning to Represent Action Values as a Hypergraph on the Action Vertices
• Learning to Dynamically Coordinate Multi-Robot Teams in Graph Attention Networks
• Simulating multi-exit evacuation using deep reinforcement learning
• A Deep Reinforcement Learning Architecture for Multi-stage Optimal Control
• Monte-Carlo Tree Search for Policy Optimization
• Measuring Progress in Deep Reinforcement Learning Sample Efficiency
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments
• A survey of benchmarking frameworks for reinforcement learning
• Instance based Generalization in Reinforcement Learning
• The Difficulty of Passive Learning in Deep Reinforcement Learning
• Correcting Momentum in Temporal Difference Learning
• Towards robust and domain agnostic reinforcement learning competitions
• Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
• Ensemble Bootstrapping for Q-Learning
• Continuous Control With Ensemble Deep Deterministic Policy Gradients
• Improving Experience Replay through Modeling of Similar Transitions' Sets
• Cascade Attribute Network: Decomposing Reinforcement Learning Control Policies using Hierarchical Neural Networks
• Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learning
• Benchmarking the Spectrum of Agent Capabilities
• Implicitly Regularized RL with Implicit Q-Values
• Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correction
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• Adversary Agnostic Robust Deep Reinforcement Learning
• Flexible and Efficient Long-Range Planning Through Curious Exploration
• Importance of using appropriate baselines for evaluation of data-efficiency in deep reinforcement learning for Atari
• Deep Reinforcement Learning with Weighted Q-Learning
• MACS: Deep Reinforcement Learning based SDN Controller Synchronization Policy Design
• Transfer Heterogeneous Knowledge Among Peer-to-Peer Teammates: A Model Distillation Approach
• Learning Gaussian Policies from Corrective Human Feedback
• Visual Explanation using Attention Mechanism in Actor-Critic-based Deep Reinforcement Learning
• Deep Reinforcement Learning for Tactile Robotics: Learning to Type on a Braille Keyboard
• Efficient decorrelation of features using Gramian in Reinforcement Learning
• Decentralized Deep Reinforcement Learning for Network Level Traffic Signal Control
• Efficient Reinforcement Learning Development with RLzoo
• Lineage Evolution Reinforcement Learning
• Marginal Contribution Feature Importance -- an Axiomatic Approach for The Natural Case
• Generalizing Decision Making for Automated Driving with an Invariant Environment Representation using Deep Reinforcement Learning
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Hybrid Supervised Reinforced Model for Dialogue Systems
• GMAC: A Distributional Perspective on Actor-Critic Framework
• Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization
• Utilizing Skipped Frames in Action Repeats via Pseudo-Actions
• Exploiting the potential of deep reinforcement learning for classification tasks in high-dimensional and unstructured data
• CubeTR: Learning to Solve The Rubiks Cube Using Transformers
• Count-Based Temperature Scheduling for Maximum Entropy Reinforcement Learning
• Neural Auto-Curricula
• Width-based Lookaheads with Learnt Base Policies and Heuristics Over the Atari-2600 Benchmark
• Crowdfunding Dynamics Tracking: A Reinforcement Learning Approach
• Planning with Expectation Models for Control
• Human-Level Control without Server-Grade Hardware
• High Performance Across Two Atari Paddle Games Using the Same Perceptual Control Architecture Without Training
• An Oracle and Observations for the OpenAI Gym / ALE Freeway Environment
• Interpretable Local Tree Surrogate Policies
• Deep reinforcement learning for RAN optimization and control
• Bayesian Distributional Policy Gradients
• Growing Action Spaces
• GrowSpace: Learning How to Shape Plants
• Playing Atari Ball Games with Hierarchical Reinforcement Learning
• Option Discovery in the Absence of Rewards with Manifold Analysis
• An adaptive synchronization approach for weights of deep reinforcement learning
• Mastering Rate based Curriculum Learning